[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 399620161 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002133] x2apic enabled [ 0.002920] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005803] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008110] LSM: Security Framework initializing [ 0.009027] Yama: becoming mindful. [ 0.010020] SELinux: Initializing. [ 0.010674] *** VALIDATE selinux *** [ 0.017083] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021325] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022081] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023060] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024055] *** VALIDATE tmpfs *** [ 0.025321] *** VALIDATE proc *** [ 0.026144] *** VALIDATE cgroup *** [ 0.027004] *** VALIDATE cgroup2 *** [ 0.028069] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031016] Spectre V2 : User space: Vulnerable [ 0.032003] Speculative Store Bypass: Vulnerable [ 0.034621] debug: unmapping init [mem 0xffffffff9ae59000-0xffffffff9ae60fff] [ 0.038131] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039417] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040010] ... version: 2 [ 0.040723] ... bit width: 48 [ 0.041017] ... generic registers: 4 [ 0.041730] ... value mask: 0000ffffffffffff [ 0.042005] ... max period: 00007fffffffffff [ 0.042932] ... fixed-purpose events: 3 [ 0.043005] ... event mask: 000000070000000f [ 0.044093] rcu: Hierarchical SRCU implementation. [ 0.045991] smp: Bringing up secondary CPUs ... [ 0.046357] x86: Booting SMP configuration: [ 0.047010] .... node #0, CPUs: #1 #2 #3 [ 0.049811] smp: Brought up 1 node, 4 CPUs [ 0.050790] smpboot: Max logical packages: 1 [ 0.051006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133013] node 0 deferred pages initialised in 81ms [ 0.136127] devtmpfs: initialized [ 0.137094] x86/mm: Memory block size: 128MB [ 0.139139] gcov: version magic: 0x41383552 [ 0.141085] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142045] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143203] pinctrl core: initialized pinctrl subsystem [ 0.144085] [ 0.144388] ************************************************************* [ 0.146008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147006] ** ** [ 0.149006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150005] ** ** [ 0.152006] ** This means that this kernel is built to expose internal ** [ 0.153005] ** IOMMU data structures, which may compromise security on ** [ 0.154005] ** your system. ** [ 0.155005] ** ** [ 0.156005] ** If you see this message and you are not debugging the ** [ 0.158006] ** kernel, report this immediately to your vendor! ** [ 0.159005] ** ** [ 0.160005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161005] ************************************************************* [ 0.163437] NET: Registered protocol family 16 [ 0.164293] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169281] cpuidle: using governor menu [ 0.170311] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171243] PCI: Using configuration type 1 for base access [ 0.172090] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.180017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182090] cryptd: max_cpu_qlen set to 1000 [ 0.184099] ACPI: Added _OSI(Module Device) [ 0.184980] ACPI: Added _OSI(Processor Device) [ 0.185006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190441] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194439] ACPI: Interpreter enabled [ 0.195035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195772] ACPI: Using IOAPIC for interrupt routing [ 0.197042] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.199297] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.206744] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.208017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.209010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.211032] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214935] acpiphp: Slot [2] registered [ 0.215075] acpiphp: Slot [5] registered [ 0.215917] acpiphp: Slot [6] registered [ 0.217054] acpiphp: Slot [7] registered [ 0.217932] acpiphp: Slot [8] registered [ 0.219050] acpiphp: Slot [9] registered [ 0.220053] acpiphp: Slot [10] registered [ 0.220851] acpiphp: Slot [3] registered [ 0.220948] acpiphp: Slot [4] registered [ 0.222067] acpiphp: Slot [11] registered [ 0.222858] acpiphp: Slot [12] registered [ 0.224044] acpiphp: Slot [13] registered [ 0.224844] acpiphp: Slot [14] registered [ 0.225042] acpiphp: Slot [15] registered [ 0.225872] acpiphp: Slot [16] registered [ 0.227044] acpiphp: Slot [17] registered [ 0.227909] acpiphp: Slot [18] registered [ 0.228044] acpiphp: Slot [19] registered [ 0.228946] acpiphp: Slot [20] registered [ 0.230042] acpiphp: Slot [21] registered [ 0.230895] acpiphp: Slot [22] registered [ 0.232054] acpiphp: Slot [23] registered [ 0.232897] acpiphp: Slot [24] registered [ 0.233042] acpiphp: Slot [25] registered [ 0.233860] acpiphp: Slot [26] registered [ 0.235054] acpiphp: Slot [27] registered [ 0.235866] acpiphp: Slot [28] registered [ 0.237068] acpiphp: Slot [29] registered [ 0.237852] acpiphp: Slot [30] registered [ 0.238050] acpiphp: Slot [31] registered [ 0.238836] PCI host bridge to bus 0000:00 [ 0.240010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.241007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.244008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245008] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.247009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.248111] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.249653] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251783] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.258011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.263009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.264008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.266008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.268300] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.270929] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.273033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.276903] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.282010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.294962] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.300007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.304364] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.316010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.328012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.348020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.359518] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.365008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.371010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.383010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.393000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.400011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.406021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.417018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.425727] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.429008] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.433021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.442014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.451359] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.457008] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.461010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.471965] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.479601] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.483007] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.487013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.496007] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.503984] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.505192] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.507200] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.508175] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.509097] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.513059] iommu: Default domain type: Passthrough [ 0.514337] SCSI subsystem initialized [ 0.515000] ACPI: bus type USB registered [ 0.516077] usbcore: registered new interface driver usbfs [ 0.518031] usbcore: registered new interface driver hub [ 0.520038] usbcore: registered new device driver usb [ 0.521112] pps_core: LinuxPPS API ver. 1 registered [ 0.523007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.525026] PTP clock support registered [ 0.527114] EDAC MC: Ver: 3.0.0 [ 0.528097] PCI: Using ACPI for IRQ routing [ 0.529573] NetLabel: Initializing [ 0.530006] NetLabel: domain hash size = 128 [ 0.530811] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.532038] NetLabel: unlabeled traffic allowed by default [ 0.533068] vgaarb: loaded [ 0.534122] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.534973] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.539207] clocksource: Switched to clocksource kvm-clock [ 0.615365] VFS: Disk quotas dquot_6.6.0 [ 0.616162] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.617543] *** VALIDATE ramfs *** [ 0.618179] *** VALIDATE hugetlbfs *** [ 0.619493] pnp: PnP ACPI init [ 0.620922] pnp: PnP ACPI: found 6 devices [ 0.633427] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.635168] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.636376] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.637487] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.638747] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.639988] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.641472] NET: Registered protocol family 2 [ 0.642956] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.645736] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.647681] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.651186] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.653101] TCP: Hash tables configured (established 65536 bind 65536) [ 0.654834] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.658763] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.661947] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.664852] NET: Registered protocol family 1 [ 0.666809] RPC: Registered named UNIX socket transport module. [ 0.669971] RPC: Registered udp transport module. [ 0.671437] RPC: Registered tcp transport module. [ 0.672890] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.675019] NET: Registered protocol family 44 [ 0.676020] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.677560] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.679037] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.680543] PCI: CLS 0 bytes, default 64 [ 0.682264] Unpacking initramfs... [ 1.821068] debug: unmapping init [mem 0xffff8c7d3cc54000-0xffff8c7d3ffbffff] [ 1.823480] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.824654] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.826246] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.258881] Initialise system trusted keyrings [ 2.259793] Key type blacklist registered [ 2.260837] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.266984] zbud: loaded [ 2.269319] *** VALIDATE nfs *** [ 2.269940] *** VALIDATE nfs4 *** [ 2.270756] pstore: using deflate compression [ 2.272277] Platform Keyring initialized [ 2.339212] NET: Registered protocol family 38 [ 2.340786] Key type asymmetric registered [ 2.342202] Asymmetric key parser 'x509' registered [ 2.343916] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.346579] io scheduler mq-deadline registered [ 2.348137] io scheduler kyber registered [ 2.349721] io scheduler bfq registered [ 2.351281] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.353972] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.356676] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.359336] ACPI: Power Button [PWRF] [ 2.364345] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.370476] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.380125] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.385893] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.397296] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.424777] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.451007] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.454520] Non-volatile memory driver v1.3 [ 2.455824] Linux agpgart interface v0.103 [ 2.479993] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.482637] vda: detected capacity change from 0 to 68628480 [ 2.494375] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.497073] vdb: detected capacity change from 0 to 1073741824 [ 2.509067] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.511764] vdc: detected capacity change from 0 to 2621440000 [ 2.523418] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.525984] vdd: detected capacity change from 0 to 2621440000 [ 2.538214] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.540394] vde: detected capacity change from 0 to 4294967296 [ 2.552431] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.554781] vdf: detected capacity change from 0 to 4294967296 [ 2.558572] libphy: Fixed MDIO Bus: probed [ 2.561579] usbcore: registered new interface driver usbserial_generic [ 2.562985] usbserial: USB Serial support registered for generic [ 2.564278] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.566599] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.567604] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.568994] mousedev: PS/2 mouse device common for all mice [ 2.570896] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.572776] rtc_cmos 00:05: RTC can wake from S4 [ 2.574439] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.575883] rtc_cmos 00:05: registered as rtc0 [ 2.577597] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.578142] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.581823] intel_pstate: CPU model not supported [ 2.583877] hid: raw HID events driver (C) Jiri Kosina [ 2.585586] usbcore: registered new interface driver usbhid [ 2.587168] usbhid: USB HID core driver [ 2.588347] drop_monitor: Initializing network drop monitor service [ 2.590199] Initializing XFRM netlink socket [ 2.591894] NET: Registered protocol family 10 [ 2.594234] Segment Routing with IPv6 [ 2.595638] NET: Registered protocol family 17 [ 2.597385] mpls_gso: MPLS GSO support [ 2.603199] RAS: Correctable Errors collector initialized. [ 2.605125] AVX version of gcm_enc/dec engaged. [ 2.606737] AES CTR mode by8 optimization enabled [ 2.651167] sched_clock: Marking stable (2651146855, 0)->(3301634870, -650488015) [ 2.653667] registered taskstats version 1 [ 2.655255] Loading compiled-in X.509 certificates [ 2.656312] zswap: loaded using pool lzo/zbud [ 2.674299] Key type big_key registered [ 2.683580] Key type encrypted registered [ 2.684458] ima: No TPM chip found, activating TPM-bypass! [ 2.685573] ima: Allocated hash algorithm: sha1 [ 2.686515] ima: No architecture policies found [ 2.687448] evm: Initialising EVM extended attributes: [ 2.688417] evm: security.selinux [ 2.689009] evm: security.ima [ 2.689636] evm: security.capability [ 2.690362] evm: HMAC attrs: 0x1 [ 2.691638] rtc_cmos 00:05: setting system clock to 2026-01-02 20:01:20 UTC (1767384080) [ 2.695439] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 2.697012] debug: unmapping init [mem 0xffffffff9ab82000-0xffffffff9ae58fff] [ 2.703069] Write protecting the kernel read-only data: 28672k [ 2.704861] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 2.706424] debug: unmapping init [mem 0xffffffff99b14000-0xffffffff99bfffff] [ 2.729049] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.733748] systemd[1]: Detected virtualization kvm. [ 2.734812] systemd[1]: Detected architecture x86-64. [ 2.735870] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.756296] systemd[1]: No hostname configured. [ 2.757168] systemd[1]: Set hostname to . [ 2.758433] random: systemd: uninitialized urandom read (16 bytes read) [ 2.759786] systemd[1]: Initializing machine ID from random generator. [ 2.791331] random: ln: uninitialized urandom read (6 bytes read) [ 2.859175] random: systemd: uninitialized urandom read (16 bytes read) [ 2.861989] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 2.865816] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 2.869924] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook...[ 3.301794] random: fast init done [ 3.348268] device-mapper: uevent: version 1.0.3 [ 3.350471] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.025434] virtio_net virtio0 ens2: renamed from eth0 [ 4.070553] scsi host0: ata_piix [ 4.120261] scsi host1: ata_piix [ 4.127090] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.129425] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.747223] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.198497] random: crng init done [ 9.199669] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.361067] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.256943] printk: systemd: 25 output lines suppressed due to ratelimiting [ 10.415171] SELinux: Disabled at runtime. [ 10.460508] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.466100] systemd[1]: Detected virtualization kvm. [ 10.467440] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.782917] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.787246] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.791237] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.796334] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.798960] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.805295] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.810566] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ 10.912570] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 11.170669] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 11.487427] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.505282] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.612198] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.632777] EDAC sbridge: Ver: 1.1.2 [ 13.039950] Key type dns_resolver registered [ 13.374138] NFS: Registering the id_resolver key type [ 13.376133] Key type id_resolver registered [ 13.377514] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-server login: [ 37.432951] libcfs: loading out-of-tree module taints kernel. [ 37.448041] Key type ._llcrypt registered [ 37.448888] Key type .llcrypt registered [ 37.482620] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 44.490924] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 45.049250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 45.056245] alg: No test for adler32 (adler32-zlib) [ 45.999643] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 46.244771] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 47.840167] Key type lgssc registered [ 48.720595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.389020] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 55.993877] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.255390] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.800241] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.703454] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.783881] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 74.329676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.357083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 74.367247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 75.461313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 75.471412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 75.507465] Lustre: lustre-MDT0000: new disk, initializing [ 75.529116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 75.534678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 76.883934] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.525388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.569333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 83.606896] Lustre: 6482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 83.622189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 83.626249] Lustre: Skipped 1 previous similar message [ 83.680712] Lustre: lustre-MDT0001: new disk, initializing [ 83.707396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 83.719860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 83.727914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 85.221354] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.809025] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 91.346298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.390074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 91.508067] Lustre: lustre-OST0000: new disk, initializing [ 91.512217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 91.535112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.006186] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.246667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 96.251619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 96.287597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 100.132157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.165444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 100.206811] Lustre: lustre-OST0001: new disk, initializing [ 100.208859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 100.227248] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.150954] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.339521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 103.343814] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 103.352210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 108.672271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 112.418133] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 119.317135] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing check_logdir /tmp/testlogs/ [ 120.768710] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing yml_node [ 122.380031] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 123.495959] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 124.365791] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 125.159596] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 2 15:03:22 EST 2026 [ 133.136993] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 133.853238] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 149.473092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.473762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 149.477488] Lustre: Skipped 2 previous similar messages [ 149.479194] Lustre: Skipped 1 previous similar message [ 152.147472] LustreError: 12185:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 152.186152] Lustre: server umount lustre-MDT0000 complete [ 153.777204] LustreError: 6473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384231 with bad export cookie 4840289726697230367 [ 153.779859] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 153.783205] LustreError: 6473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 154.592977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.593296] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 154.602066] Lustre: Skipped 2 previous similar messages [ 154.604194] Lustre: Skipped 2 previous similar messages [ 158.625198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 158.628788] Lustre: Skipped 2 previous similar messages [ 159.951543] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 159.955477] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 160.021122] Lustre: server umount lustre-MDT0001 complete [ 161.455127] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 161.457731] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 161.484682] Lustre: server umount lustre-OST0000 complete [ 162.689371] Lustre: server umount lustre-OST0001 complete [ 165.116701] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 168.295282] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 172.015941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 174.292315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.929232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.647064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.595848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.636680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.785896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.808376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.864770] Lustre: lustre-MDT0000: new disk, initializing [ 182.892787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.899737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.569911] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 189.764059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.795773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 189.848200] Lustre: 14967:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.991681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.293760] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 194.774969] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 196.226949] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 199.079490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.107691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 199.202069] Lustre: lustre-OST0000: new disk, initializing [ 199.204403] Lustre: Skipped 1 previous similar message [ 199.207054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.209416] Lustre: Skipped 2 previous similar messages [ 199.241664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.245070] Lustre: Skipped 1 previous similar message [ 200.892691] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 201.023125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.025527] Lustre: Skipped 1 previous similar message [ 201.027183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.062060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.144843] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 205.877405] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 205.945401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 207.085861] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 207.150179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 211.425977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 211.427937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 211.428806] LustreError: Skipped 1 previous similar message [ 211.434268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 211.437160] Lustre: Skipped 1 previous similar message [ 214.068118] LustreError: 17908:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 214.073475] LustreError: 17908:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 214.140559] Lustre: server umount lustre-OST0000 complete [ 216.547903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 216.549362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 216.558973] Lustre: Skipped 2 previous similar messages [ 216.565709] Lustre: Skipped 1 previous similar message [ 226.785281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 226.787336] Lustre: Skipped 3 previous similar messages [ 230.368108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 230.401667] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 230.403360] LustreError: 18109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 230.429681] Lustre: server umount lustre-MDT0000 complete [ 231.592291] LustreError: 15749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384309 with bad export cookie 4840289726697231886 [ 231.596518] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 231.597466] LustreError: 15749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 231.712279] Lustre: server umount lustre-MDT0001 complete [ 232.833185] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:05:10 (1767384310) [ 234.973318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.116351] LustreError: 18836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.122444] LustreError: 18836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 235.138244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.312941] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 239.292245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 240.573466] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 242.237371] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 243.443938] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 245.695537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 247.569947] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 249.272549] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 270.306759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.307562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 270.310679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 270.311836] LustreError: Skipped 1 previous similar message [ 270.316664] Lustre: Skipped 1 previous similar message [ 273.654129] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 273.656629] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 273.689847] Lustre: server umount lustre-OST0000 complete [ 275.426157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 289.248090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 289.309673] Lustre: server umount lustre-MDT0000 complete [ 290.500424] LustreError: 19554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384368 with bad export cookie 4840289726697232768 [ 290.504281] LustreError: 19554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 290.504710] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.569368] LustreError: 21509:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 290.571877] LustreError: 21509:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 290.621523] Lustre: server umount lustre-MDT0001 complete [ 293.935621] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 294.997859] Key type lgssc unregistered [ 295.129339] LNet: 22039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.131809] LNetError: 22039:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.140375] LNet: Removed LNI 192.168.203.106@tcp [ 295.483109] Key type .llcrypt unregistered [ 295.484365] Key type ._llcrypt unregistered [ 304.315913] Key type ._llcrypt registered [ 304.317315] Key type .llcrypt registered [ 304.355234] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 304.648356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 304.661053] alg: No test for adler32 (adler32-zlib) [ 305.523227] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 305.618404] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 307.200196] Key type lgssc registered [ 307.637598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 310.513299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 310.522291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 311.693651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 313.095164] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 316.509378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 316.622467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 317.947052] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 319.654853] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 320.747424] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 322.757514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 322.880941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.849252] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 326.645843] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 336.231368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 342.497707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 342.499791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.503317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 346.594987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 346.595844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 346.595890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.602266] Lustre: Skipped 1 previous similar message [ 351.716088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.718024] Lustre: Skipped 1 previous similar message [ 356.320094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 356.337050] LustreError: 26120:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 356.371095] Lustre: server umount lustre-OST0000 complete [ 357.857191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.861514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.869890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.953560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.954275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.192121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 372.193548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.198041] Lustre: Skipped 4 previous similar messages [ 372.232071] LustreError: 26323:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 372.233888] LustreError: 26323:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 372.265155] Lustre: server umount lustre-MDT0000 complete [ 373.426588] LustreError: 23390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384451 with bad export cookie 182349650369533388 [ 373.429515] LustreError: 23390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 373.430620] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.501678] LustreError: 26528:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 373.503327] LustreError: 26528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 373.557194] Lustre: server umount lustre-MDT0001 complete [ 376.993121] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 378.044770] Key type lgssc unregistered [ 378.168386] LNet: 27059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.171629] LNetError: 27059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.178337] LNet: Removed LNI 192.168.203.106@tcp [ 378.468213] Key type .llcrypt unregistered [ 378.469365] Key type ._llcrypt unregistered [ 387.234722] Key type ._llcrypt registered [ 387.237067] Key type .llcrypt registered [ 387.279967] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 387.618360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 387.652076] alg: No test for adler32 (adler32-zlib) [ 388.493091] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 388.571795] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 390.152200] Key type lgssc registered [ 390.569682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 393.132864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 393.141151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.301315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.548655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 398.366547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 398.486822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.705584] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 401.342694] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 402.471247] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 407.078782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 407.192923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.980570] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 410.686180] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 419.497795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 423.905269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.908736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.915255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.932597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.932808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.933521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.952910] Lustre: Skipped 1 previous similar message [ 428.527831] LustreError: 31041:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 428.569950] Lustre: server umount lustre-OST0000 complete [ 430.051236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.060647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 430.064088] Lustre: Skipped 1 previous similar message [ 435.169945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.173784] Lustre: Skipped 1 previous similar message [ 440.290652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.294262] Lustre: Skipped 1 previous similar message [ 444.384190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.429680] LustreError: 31243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 444.432064] LustreError: 31243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 444.466172] Lustre: server umount lustre-MDT0000 complete [ 445.409142] LustreError: 28430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.414102] LustreError: 28430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 445.842647] LustreError: 28411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384523 with bad export cookie 18199975366142010305 [ 445.847050] LustreError: 28411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 445.847862] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.925514] LustreError: 31446:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 445.927568] LustreError: 31446:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 445.975212] Lustre: server umount lustre-MDT0001 complete [ 449.372621] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 450.704768] Key type lgssc unregistered [ 450.864605] LNet: 31978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.869957] LNetError: 31978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.880468] LNet: Removed LNI 192.168.203.106@tcp [ 451.176506] Key type .llcrypt unregistered [ 451.178035] Key type ._llcrypt unregistered [ 460.952904] Key type ._llcrypt registered [ 460.953830] Key type .llcrypt registered [ 460.989291] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 461.284299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 461.340065] alg: No test for adler32 (adler32-zlib) [ 462.199606] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 462.284113] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 463.872169] Key type lgssc registered [ 464.236883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 467.131330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 467.137411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.303786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.817956] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 473.015793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 473.149496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 474.426777] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 476.744477] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 478.065993] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 480.333260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 480.434669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.193301] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 484.376545] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 494.764694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 498.145449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.150063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.157953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.193346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.195086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.195553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.201083] Lustre: Skipped 1 previous similar message [ 503.799190] LustreError: 35618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 503.849687] Lustre: server umount lustre-OST0000 complete [ 505.313252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.313931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 505.316428] Lustre: Skipped 1 previous similar message [ 510.433328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.435697] Lustre: Skipped 2 previous similar messages [ 511.235673] LustreError: 35819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 511.239213] LustreError: 35819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 511.268401] Lustre: server umount lustre-MDT0000 complete [ 512.643448] LustreError: 35518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384590 with bad export cookie 2267742394852646698 [ 512.646067] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.649104] LustreError: 35518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 512.723535] LustreError: 36021:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 512.725475] LustreError: 36021:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 512.772626] Lustre: server umount lustre-MDT0001 complete [ 516.533328] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 517.504830] Key type lgssc unregistered [ 517.634589] LNet: 36551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.638602] LNetError: 36551:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.646522] LNet: Removed LNI 192.168.203.106@tcp [ 517.987509] Key type .llcrypt unregistered [ 517.988906] Key type ._llcrypt unregistered [ 527.091533] Key type ._llcrypt registered [ 527.092921] Key type .llcrypt registered [ 527.129188] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 527.571081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 527.596899] alg: No test for adler32 (adler32-zlib) [ 528.448589] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 528.538134] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 530.112247] Key type lgssc registered [ 530.576754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 533.685434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 533.693476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.896171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.248344] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 539.749685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 539.916266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 541.235903] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 543.139648] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 544.393157] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 546.897213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.054543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.153469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 551.364735] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 559.339107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 564.707086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.709164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.710478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.715837] Lustre: Skipped 1 previous similar message [ 567.127592] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 567.129306] Lustre: Skipped 1 previous similar message [ 569.831663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.836561] Lustre: Skipped 1 previous similar message [ 570.288194] LustreError: 40184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 570.325692] Lustre: server umount lustre-OST0000 complete [ 583.483165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.484822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.488918] Lustre: Skipped 1 previous similar message [ 583.493573] Lustre: Skipped 1 previous similar message [ 588.770324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.772190] Lustre: Skipped 1 previous similar message [ 589.645600] LustreError: 40559:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 589.649222] LustreError: 40559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 589.693201] Lustre: server umount lustre-MDT0000 complete [ 590.968274] LustreError: 37901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384668 with bad export cookie 3722057168120562457 [ 590.972595] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.120648] LustreError: 40762:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 597.122766] LustreError: 40762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 597.183739] Lustre: server umount lustre-MDT0001 complete [ 600.745254] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 601.914699] Key type lgssc unregistered [ 602.043426] LNet: 41293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.046396] LNetError: 41293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.053548] LNet: Removed LNI 192.168.203.106@tcp [ 602.403182] Key type .llcrypt unregistered [ 602.405374] Key type ._llcrypt unregistered [ 612.111175] Key type ._llcrypt registered [ 612.112082] Key type .llcrypt registered [ 612.150947] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 612.595794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.610484] alg: No test for adler32 (adler32-zlib) [ 613.486812] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 613.585670] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 615.168154] Key type lgssc registered [ 615.683292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 619.361069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 619.376054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.539298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.763645] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 624.399414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 624.524042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 625.780767] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 627.539491] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 628.706790] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 630.955452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 631.075124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.945173] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 634.653479] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 645.418828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 655.187601] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 655.328913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 655.331206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.842847] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.844213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 655.846428] Lustre: Skipped 1 previous similar message [ 655.848763] Lustre: Skipped 1 previous similar message [ 658.428919] LustreError: 44931:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 658.452887] Lustre: server umount lustre-MDT0000 complete [ 659.665937] LustreError: 44283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384737 with bad export cookie 4327910442093187389 [ 659.669112] LustreError: 44283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.669402] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.299458] LustreError: 42660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.303644] LustreError: 42660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 660.306500] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 660.308208] Lustre: Skipped 1 previous similar message [ 660.960924] LustreError: 43375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.961240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.992863] LustreError: 43375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.993184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 664.998038] Lustre: Skipped 1 previous similar message [ 665.790589] LustreError: 45132:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 665.792574] LustreError: 45132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 665.839019] Lustre: server umount lustre-MDT0001 complete [ 683.564150] LustreError: 45357:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 683.565821] LustreError: 45357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 683.599341] Lustre: server umount lustre-OST0000 complete [ 688.626662] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 689.690517] Key type lgssc unregistered [ 689.821460] LNet: 46195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.825582] LNetError: 46195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.835453] LNet: Removed LNI 192.168.203.106@tcp [ 690.106073] Key type .llcrypt unregistered [ 690.107076] Key type ._llcrypt unregistered [ 699.749184] Key type ._llcrypt registered [ 699.750322] Key type .llcrypt registered [ 699.791975] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 700.355497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.366540] alg: No test for adler32 (adler32-zlib) [ 701.223967] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 701.312923] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 702.896169] Key type lgssc registered [ 703.292434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 705.838306] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 705.848731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 721.312423] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 726.432226] LustreError: 47538:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 753.056187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.940450] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 756.645122] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.660061] LustreError: 48246:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 920.695349] Lustre: server umount lustre-OST0000 complete [ 925.883234] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 926.846930] Key type lgssc unregistered [ 926.972322] LNet: 49080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.976557] LNetError: 49080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.987266] LNet: Removed LNI 192.168.203.106@tcp [ 927.266728] Key type .llcrypt unregistered [ 927.268070] Key type ._llcrypt unregistered [ 936.409225] Key type ._llcrypt registered [ 936.410315] Key type .llcrypt registered [ 936.461994] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 936.788276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.820282] alg: No test for adler32 (adler32-zlib) [ 937.673811] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 937.755848] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 939.344177] Key type lgssc registered [ 939.821359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 942.635301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 942.641292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.807477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.072220] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 948.236176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.386485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 949.707396] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 951.578845] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 952.716068] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 954.954871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 955.087093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.797975] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 958.473814] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 964.577961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.578533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.585355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.359380] LustreError: 52692:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 965.400573] Lustre: server umount lustre-OST0000 complete [ 969.184960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 969.187653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.195174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.674525] LustreError: 52898:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 972.678168] LustreError: 52898:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 972.704675] Lustre: server umount lustre-MDT0000 complete [ 973.992269] LustreError: 50436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385051 with bad export cookie 17017896047390295046 [ 973.996122] LustreError: 50436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 973.996322] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.065108] LustreError: 53102:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 974.066715] LustreError: 53102:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 974.126918] Lustre: server umount lustre-MDT0001 complete [ 977.698841] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 978.660779] Key type lgssc unregistered [ 978.778381] LNet: 53633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.780789] LNetError: 53633:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.792321] LNet: Removed LNI 192.168.203.106@tcp [ 979.093696] Key type .llcrypt unregistered [ 979.095413] Key type ._llcrypt unregistered [ 987.833379] Key type ._llcrypt registered [ 987.834376] Key type .llcrypt registered [ 987.876506] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 988.276794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.339188] alg: No test for adler32 (adler32-zlib) [ 989.235639] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 989.325037] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 990.912163] Key type lgssc registered [ 991.317706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.830515] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.836186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1009.312362] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1014.432203] LustreError: 54978:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 1041.056227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.597535] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1045.460693] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1048.309982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.568481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.604371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1050.136857] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1053.667708] LustreError: 55005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.678919] LustreError: 55005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1054.124365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1054.302459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.860020] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1057.752939] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.887820] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1060.836110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.836793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1060.841739] Lustre: Skipped 1 previous similar message [ 1065.648195] LustreError: 57260:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1065.704875] Lustre: server umount lustre-OST0000 complete [ 1084.897249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1084.901080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.908598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.911775] Lustre: Skipped 1 previous similar message [ 1086.593624] LustreError: 57638:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1086.597691] LustreError: 57638:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1086.630123] Lustre: server umount lustre-MDT0000 complete [ 1088.101148] LustreError: 56634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385165 with bad export cookie 7870423730311591081 [ 1088.107346] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.108842] LustreError: 56634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1094.292826] LustreError: 57839:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1094.297179] LustreError: 57839:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1094.364601] Lustre: server umount lustre-MDT0001 complete [ 1098.092620] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1099.557610] Key type lgssc unregistered [ 1099.721570] LNet: 58370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.726730] LNetError: 58370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.738554] LNet: Removed LNI 192.168.203.106@tcp [ 1100.215850] Key type .llcrypt unregistered [ 1100.218118] Key type ._llcrypt unregistered [ 1110.075367] Key type ._llcrypt registered [ 1110.076718] Key type .llcrypt registered [ 1110.125209] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1110.498264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.519056] alg: No test for adler32 (adler32-zlib) [ 1111.383782] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1111.467974] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1113.048180] Key type lgssc registered [ 1113.401575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.472455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.483992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.680878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.321316] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1122.274255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1122.437900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.882347] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1125.984914] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1127.143800] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1130.279065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1130.470769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.581137] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1134.927510] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1142.761055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1147.877227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.878561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1147.879207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1147.883183] Lustre: Skipped 1 previous similar message [ 1147.887356] LustreError: Skipped 1 previous similar message [ 1149.553118] LustreError: 62009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1149.594889] Lustre: server umount lustre-OST0000 complete [ 1152.995233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.006296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.011150] Lustre: Skipped 2 previous similar messages [ 1153.230379] LustreError: 62212:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1153.234635] LustreError: 62212:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1153.264398] Lustre: server umount lustre-MDT0000 complete [ 1154.920592] LustreError: 61361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385232 with bad export cookie 4639859549759024262 [ 1154.926116] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.927505] LustreError: 61361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1155.016706] LustreError: 62413:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1155.018976] LustreError: 62413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1155.071842] Lustre: server umount lustre-MDT0001 complete [ 1159.418175] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1160.891081] Key type lgssc unregistered [ 1161.027476] LNet: 62944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.031225] LNetError: 62944:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.038399] LNet: Removed LNI 192.168.203.106@tcp [ 1161.417734] Key type .llcrypt unregistered [ 1161.418837] Key type ._llcrypt unregistered [ 1171.763980] Key type ._llcrypt registered [ 1171.765319] Key type .llcrypt registered [ 1171.802098] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1172.095116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.101256] alg: No test for adler32 (adler32-zlib) [ 1172.949709] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1173.041688] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1174.624225] Key type lgssc registered [ 1175.196643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.837624] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 15:20:55 (1767385255) [ 1181.547161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1181.555117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.709449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.895344] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1187.200220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1187.334465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.655709] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1190.703116] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1192.052765] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1195.023748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1195.237222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.742747] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1199.821653] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1208.557298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1212.874124] Lustre: Failing over lustre-MDT0000 [ 1212.899056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1212.903593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.910411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.996849] LustreError: 66872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1213.029175] Lustre: server umount lustre-MDT0000 complete [ 1213.922762] LustreError: 64606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.925271] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.929279] LustreError: 64606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1213.934779] Lustre: Skipped 1 previous similar message [ 1215.321285] LustreError: 65880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1216.968505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.022720] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.113682] Lustre: *** cfs_fail_loc=135, val=0*** [ 1217.163218] LustreError: 67382:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1217.167644] LustreError: 67382:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -2 [ 1217.173226] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1217.173226] [ 1217.178922] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1217.184759] LustreError: 67369:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1217.188185] LustreError: 67369:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -2 [ 1217.191746] LustreError: 67369:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 1217.194433] LustreError: 67369:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1217.209448] Lustre: server umount lustre-MDT0000 complete [ 1217.210902] LustreError: 67369:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1219.042933] LustreError: 64606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.050754] LustreError: 64606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1220.181252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.220692] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.333582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.354940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1220.427155] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1221.932909] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1225.707475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1225.720625] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1225.747747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1228.257175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.260970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.267173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.822623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.824983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.828578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.076153] LustreError: 68194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1233.080391] LustreError: 68194:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1233.140716] Lustre: server umount lustre-OST0000 complete [ 1235.939212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.939364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1235.940330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.940338] Lustre: Skipped 1 previous similar message [ 1235.946759] Lustre: Skipped 1 previous similar message [ 1241.057401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.061228] Lustre: Skipped 2 previous similar messages [ 1241.155575] LustreError: 68395:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1241.158353] LustreError: 68395:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1241.190701] Lustre: server umount lustre-MDT0000 complete [ 1243.029364] LustreError: 64587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385320 with bad export cookie 17722957153254666823 [ 1243.034692] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.037685] LustreError: 64587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1243.190282] Lustre: server umount lustre-MDT0001 complete [ 1246.600947] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1247.442816] Key type lgssc unregistered [ 1247.563367] LNet: 69127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.566396] LNetError: 69127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.577421] LNet: Removed LNI 192.168.203.106@tcp [ 1247.890529] Key type .llcrypt unregistered [ 1247.893559] Key type ._llcrypt unregistered [ 1258.238966] Key type ._llcrypt registered [ 1258.240733] Key type .llcrypt registered [ 1258.291324] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1258.653853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1258.703278] alg: No test for adler32 (adler32-zlib) [ 1259.551306] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1259.623742] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1261.200234] Key type lgssc registered [ 1261.774351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1265.015899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.025678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.207776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.943533] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1271.564450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1271.750185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1273.556324] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1275.775539] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1277.088993] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1280.374430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1280.593248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.076715] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1285.248579] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1290.870060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1294.001305] Lustre: Failing over lustre-MDT0000 [ 1294.224059] LustreError: 72745:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1294.254484] Lustre: server umount lustre-MDT0000 complete [ 1296.355837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.359693] LustreError: 70492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.360053] Lustre: Skipped 2 previous similar messages [ 1296.365230] LustreError: 70492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1298.756445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.819880] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.905882] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1298.991441] LustreError: 73257:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1298.995787] LustreError: 73257:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -9 [ 1299.002120] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1299.002120] [ 1299.007771] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1299.015756] LustreError: 73244:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1299.020372] LustreError: 73244:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -9 [ 1299.026274] LustreError: 73244:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 1299.029854] LustreError: 73244:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1299.042324] Lustre: server umount lustre-MDT0000 complete [ 1299.043866] LustreError: 73244:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1301.474742] LustreError: 70498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.481294] LustreError: 70498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1302.010747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.071980] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.212869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.246262] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1304.037449] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1306.955614] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1307.630235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1307.642358] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1307.667984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1312.740561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1312.746076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.750932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.753376] Lustre: Skipped 1 previous similar message [ 1314.994117] LustreError: 74072:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1314.997572] LustreError: 74072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1315.039640] Lustre: server umount lustre-OST0000 complete [ 1317.858864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.859143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.860284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.860290] Lustre: Skipped 1 previous similar message [ 1317.864481] Lustre: Skipped 1 previous similar message [ 1317.881554] LustreError: Skipped 1 previous similar message [ 1322.977290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.979788] Lustre: Skipped 2 previous similar messages [ 1328.097054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.099097] Lustre: Skipped 1 previous similar message [ 1331.168200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1331.206139] LustreError: 74274:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1331.209332] LustreError: 74274:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1331.238434] Lustre: server umount lustre-MDT0000 complete [ 1332.194348] LustreError: 70498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.204482] LustreError: 70498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1332.642395] LustreError: 72748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385410 with bad export cookie 17866293408900896455 [ 1332.646195] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.647434] LustreError: 72748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1332.778839] Lustre: server umount lustre-MDT0001 complete [ 1336.711738] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1337.908712] Key type lgssc unregistered [ 1338.065656] LNet: 75007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.070317] LNetError: 75007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.080609] LNet: Removed LNI 192.168.203.106@tcp [ 1338.584129] Key type .llcrypt unregistered [ 1338.586335] Key type ._llcrypt unregistered [ 1343.184476] Key type ._llcrypt registered [ 1343.185388] Key type .llcrypt registered [ 1343.219390] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1343.301522] Key type .llcrypt unregistered [ 1343.302486] Key type ._llcrypt unregistered [ 1353.200366] Key type ._llcrypt registered [ 1353.201422] Key type .llcrypt registered [ 1353.248615] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1353.617722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1353.634290] alg: No test for adler32 (adler32-zlib) [ 1354.485000] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1354.571664] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1356.160180] Key type lgssc registered [ 1356.697385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.173057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.184811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.361335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.875232] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1366.292338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1366.444990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.686968] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1369.504755] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1370.718204] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1373.495583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1373.611718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.431100] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1377.693486] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1386.925200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1392.101709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.103701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1392.112782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1392.116065] LustreError: Skipped 1 previous similar message [ 1397.218375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1397.220683] Lustre: Skipped 2 previous similar messages [ 1397.298364] LustreError: 79128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1397.343366] Lustre: server umount lustre-OST0000 complete [ 1402.339011] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1402.339804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.340869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.358563] Lustre: Skipped 2 previous similar messages [ 1405.131352] LustreError: 79329:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1405.135322] LustreError: 79329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1405.170483] Lustre: server umount lustre-MDT0000 complete [ 1406.506831] LustreError: 79331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385484 with bad export cookie 4428704643314747774 [ 1406.512901] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.516838] LustreError: 79331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1406.616964] LustreError: 79532:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1406.619559] LustreError: 79532:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1406.666487] Lustre: server umount lustre-MDT0001 complete [ 1410.870505] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1412.141822] Key type lgssc unregistered [ 1412.262384] LNet: 80063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.265271] LNetError: 80063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.277319] LNet: Removed LNI 192.168.203.106@tcp [ 1412.528404] Key type .llcrypt unregistered [ 1412.529416] Key type ._llcrypt unregistered [ 1422.837405] Key type ._llcrypt registered [ 1422.839269] Key type .llcrypt registered [ 1422.892022] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1423.385974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.391805] alg: No test for adler32 (adler32-zlib) [ 1424.266810] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1424.361719] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1425.960214] Key type lgssc registered [ 1426.402309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1429.734910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1429.742808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.934211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.239607] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1435.710720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1435.892727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1437.575889] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1440.204431] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1441.885091] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1444.927074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1445.138083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.963801] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1450.528189] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1455.403632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1460.714634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.714723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1460.725559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1460.731463] Lustre: Skipped 1 previous similar message [ 1464.499251] LustreError: 83680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1464.562306] Lustre: server umount lustre-OST0000 complete [ 1466.849457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1466.854687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.859529] Lustre: Skipped 1 previous similar message [ 1466.861733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.946871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.948073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.657880] LustreError: 83882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1472.662534] LustreError: 83882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1472.704068] Lustre: server umount lustre-MDT0000 complete [ 1474.382054] LustreError: 83053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385552 with bad export cookie 13432778721848609887 [ 1474.386880] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.388514] LustreError: 83053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1474.463626] LustreError: 84084:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1474.465870] LustreError: 84084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1474.523825] Lustre: server umount lustre-MDT0001 complete [ 1479.268339] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1480.909169] Key type lgssc unregistered [ 1481.086706] LNet: 84615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.092569] LNetError: 84615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.104813] LNet: Removed LNI 192.168.203.106@tcp [ 1481.573054] Key type .llcrypt unregistered [ 1481.575355] Key type ._llcrypt unregistered [ 1491.265809] Key type ._llcrypt registered [ 1491.266940] Key type .llcrypt registered [ 1491.329846] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1491.801137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.835054] alg: No test for adler32 (adler32-zlib) [ 1492.728113] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1492.832366] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1494.448319] Key type lgssc registered [ 1494.992367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1498.854509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1498.866963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.118192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.863907] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1506.121767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1506.335543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1508.060049] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1510.339883] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1511.999579] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1515.283633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1515.486745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.315723] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1520.950721] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1524.726745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1531.874653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1531.879498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.888115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.954205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.955890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.957709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.969074] LustreError: Skipped 1 previous similar message [ 1536.244176] LustreError: 88231:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1536.305788] Lustre: server umount lustre-OST0000 complete [ 1540.068642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.070030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1540.076220] Lustre: Skipped 1 previous similar message [ 1540.079021] Lustre: Skipped 1 previous similar message [ 1544.338806] LustreError: 88434:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1544.343370] LustreError: 88434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1544.392721] Lustre: server umount lustre-MDT0000 complete [ 1545.186055] LustreError: 85986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.190940] LustreError: 85986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1546.007072] LustreError: 85965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385623 with bad export cookie 1784885301349858863 [ 1546.010738] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1546.010800] LustreError: 85965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1546.106820] LustreError: 88637:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1546.109579] LustreError: 88637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1546.165459] Lustre: server umount lustre-MDT0001 complete [ 1550.153568] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1551.407377] Key type lgssc unregistered [ 1551.563675] LNet: 89168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.565747] LNetError: 89168:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.574597] LNet: Removed LNI 192.168.203.106@tcp [ 1552.044636] Key type .llcrypt unregistered [ 1552.047097] Key type ._llcrypt unregistered [ 1562.356605] Key type ._llcrypt registered [ 1562.357892] Key type .llcrypt registered [ 1562.413253] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1562.797583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.825102] alg: No test for adler32 (adler32-zlib) [ 1563.751439] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1563.874589] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1565.472217] Key type lgssc registered [ 1566.072332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1569.571730] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1569.582836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1585.056275] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1590.176222] LustreError: 90510:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 1616.800200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.400383] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1621.158227] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1623.868846] LustreError: 91368:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1623.919475] Lustre: server umount lustre-OST0000 complete [ 1627.451384] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:28:24 (1767385704) [ 1630.530783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.725959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.954759] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1635.806876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1635.811414] LustreError: 92038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.824450] LustreError: 92038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1635.956448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.590700] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1639.631819] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1641.337995] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1643.903747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1644.090799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.903463] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1649.506243] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1655.351175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1665.509816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.511002] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1665.511660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1665.512831] Lustre: Skipped 1 previous similar message [ 1670.382101] LustreError: 94463:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1670.384150] LustreError: 94463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1670.416204] Lustre: server umount lustre-OST0000 complete [ 1671.649884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.654796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.663052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.666602] Lustre: Skipped 1 previous similar message [ 1675.745134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.745533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.755552] LustreError: 94666:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1677.757111] LustreError: 94666:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1677.779765] Lustre: server umount lustre-MDT0000 complete [ 1679.238938] LustreError: 92756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385757 with bad export cookie 5344910282358567638 [ 1679.244322] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.245202] LustreError: 92756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1679.370935] Lustre: server umount lustre-MDT0001 complete [ 1683.470064] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1684.653398] Key type lgssc unregistered [ 1684.795646] LNet: 95401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.800067] LNetError: 95401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.808566] LNet: Removed LNI 192.168.203.106@tcp [ 1685.197550] Key type .llcrypt unregistered [ 1685.198955] Key type ._llcrypt unregistered [ 1694.478481] Key type ._llcrypt registered [ 1694.479419] Key type .llcrypt registered [ 1694.517707] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1694.890415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1694.925326] alg: No test for adler32 (adler32-zlib) [ 1695.775889] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1695.869906] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1697.456251] Key type lgssc registered [ 1697.809406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1700.380488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1700.388854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1701.581660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1703.001078] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1705.870961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1706.064828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1707.649765] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1709.814154] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1710.843487] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1713.820978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1713.981819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.369835] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1718.531456] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1726.320511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1736.685123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1736.685190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.691729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1736.701086] Lustre: Skipped 1 previous similar message [ 1736.708330] Lustre: Skipped 1 previous similar message [ 1741.800722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.805719] Lustre: Skipped 1 previous similar message [ 1746.400157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1746.422149] LustreError: 99040:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1746.474363] Lustre: server umount lustre-OST0000 complete [ 1752.034742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.034816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.035617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.036572] LustreError: Skipped 1 previous similar message [ 1752.040868] Lustre: Skipped 1 previous similar message [ 1754.388792] LustreError: 99243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1754.392720] LustreError: 99243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1754.430363] Lustre: server umount lustre-MDT0000 complete [ 1756.099605] LustreError: 99244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385833 with bad export cookie 1185150864947800857 [ 1756.103184] LustreError: 99244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1756.103759] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.190502] LustreError: 99445:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1756.193633] LustreError: 99445:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1756.271502] Lustre: server umount lustre-MDT0001 complete [ 1759.781673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1761.069290] Key type lgssc unregistered [ 1761.224554] LNet: 99976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.227400] LNetError: 99976:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.235718] LNet: Removed LNI 192.168.203.106@tcp [ 1761.631718] Key type .llcrypt unregistered [ 1761.633476] Key type ._llcrypt unregistered [ 1770.284447] Key type ._llcrypt registered [ 1770.287393] Key type .llcrypt registered [ 1770.363897] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1770.770452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.790049] alg: No test for adler32 (adler32-zlib) [ 1771.711072] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1771.822891] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1773.440213] Key type lgssc registered [ 1774.057429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1777.205355] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1777.210739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1792.672364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1797.792236] LustreError: 101176:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 1824.416214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.882932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1829.274757] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1832.551420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.689805] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1832.698477] LustreError: 102087:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1832.702858] LustreError: 102087:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -2 [ 1832.707709] LustreError: 102087:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 1832.745549] Lustre: server umount lustre-MDT0000 complete [ 1832.748887] LustreError: 102087:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1846.752913] LustreError: 102518:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1846.810305] Lustre: server umount lustre-OST0000 complete [ 1849.689212] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 1853.347937] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1857.875371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1860.871844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1863.854235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1866.976226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1870.298112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1870.344236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.498330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1870.518815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1870.575522] Lustre: lustre-MDT0000: new disk, initializing [ 1870.619872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.627075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1872.509651] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1878.260081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1878.298158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.361838] Lustre: 104851:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1878.395954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1878.398510] Lustre: Skipped 1 previous similar message [ 1878.439546] Lustre: lustre-MDT0001: new disk, initializing [ 1878.484923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.506761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1878.512088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1880.378624] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1884.143562] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1885.417429] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1888.701055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1888.745746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1888.916696] Lustre: lustre-OST0000: new disk, initializing [ 1888.920672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1888.974321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.842996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1890.850824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1890.926366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1891.772663] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1895.824412] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1897.871781] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1898.017387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1899.828218] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1899.942679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1901.032104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.033313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.037138] LustreError: Skipped 1 previous similar message [ 1901.046708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.053476] Lustre: Skipped 1 previous similar message [ 1906.151462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.155642] Lustre: Skipped 1 previous similar message [ 1911.273488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.277016] Lustre: Skipped 1 previous similar message [ 1915.360160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1915.387273] LustreError: 107794:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1915.392845] LustreError: 107794:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1915.445476] Lustre: server umount lustre-OST0000 complete [ 1919.457867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1919.463875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.473061] Lustre: Skipped 1 previous similar message [ 1919.477145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.508677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.406226] LustreError: 107997:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1923.410310] LustreError: 107997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1923.448009] Lustre: server umount lustre-MDT0000 complete [ 1924.931707] LustreError: 104842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386002 with bad export cookie 18344485282549481152 [ 1924.934785] LustreError: 104842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1924.935398] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.070690] Lustre: server umount lustre-MDT0001 complete [ 1928.655644] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:33:25 (1767386005) [ 1930.182205] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1937.258645] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 1940.863740] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1944.611457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1947.144896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1949.933960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1952.613988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1955.644455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1955.694233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.842053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1955.862541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1955.920539] Lustre: lustre-MDT0000: new disk, initializing [ 1955.964444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.977417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1957.931459] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1963.491819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1963.537156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1963.597108] Lustre: 111329:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1963.634449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1963.637555] Lustre: Skipped 1 previous similar message [ 1963.698100] Lustre: lustre-MDT0001: new disk, initializing [ 1963.806368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1963.813391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1965.637234] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1969.134938] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1970.342009] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.658712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.682495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1972.781348] Lustre: lustre-OST0000: new disk, initializing [ 1972.783671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1972.816145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.820627] Lustre: Skipped 1 previous similar message [ 1975.243852] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1976.324439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1976.330057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1976.384401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1979.116379] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1980.929571] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1980.999467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1982.755715] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1982.877089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1986.532545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.533258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.536307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.540203] Lustre: Skipped 1 previous similar message [ 1986.543482] Lustre: Skipped 3 previous similar messages [ 1989.679094] LustreError: 114266:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1989.681995] LustreError: 114266:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1989.716508] Lustre: server umount lustre-OST0000 complete [ 1991.651443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.658909] Lustre: Skipped 1 previous similar message [ 1996.770291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.774393] Lustre: Skipped 3 previous similar messages [ 1997.014635] LustreError: 114468:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1997.018208] LustreError: 114468:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1997.052219] Lustre: server umount lustre-MDT0000 complete [ 1998.248134] LustreError: 111320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386076 with bad export cookie 18344485282549482034 [ 1998.250540] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.251032] LustreError: 111320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1998.353613] Lustre: server umount lustre-MDT0001 complete [ 2002.254909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.536851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.173929] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2008.034120] LustreError: 115146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.043487] LustreError: 115146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2008.216203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2010.136760] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2012.685382] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2014.372503] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2017.558263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2020.178739] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2022.782271] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2031.222790] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2033.124673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.129564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2033.130179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.135253] Lustre: Skipped 1 previous similar message [ 2033.139862] LustreError: Skipped 1 previous similar message [ 2034.547881] LustreError: 117490:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2034.552142] LustreError: 117490:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2034.608287] Lustre: server umount lustre-OST0000 complete [ 2038.576349] Lustre: server umount lustre-MDT0000 complete [ 2040.319848] LustreError: 117403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386118 with bad export cookie 18344485282549482916 [ 2040.325175] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.326031] LustreError: 117403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2040.460843] Lustre: server umount lustre-MDT0001 complete [ 2044.647114] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2046.129673] Key type lgssc unregistered [ 2046.309528] LNet: 118425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.316247] LNetError: 118425:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.329675] LNet: Removed LNI 192.168.203.106@tcp [ 2046.784549] Key type .llcrypt unregistered [ 2046.786388] Key type ._llcrypt unregistered [ 2054.379236] Key type ._llcrypt registered [ 2054.381113] Key type .llcrypt registered [ 2054.444465] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 2062.427530] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2062.844449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.881941] alg: No test for adler32 (adler32-zlib) [ 2063.741320] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2063.836734] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2065.424212] Key type lgssc registered [ 2066.025713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.420680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2073.405335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2075.898341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2079.066768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2082.217919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2082.239104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2082.246962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.355273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2083.365956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2083.404249] Lustre: lustre-MDT0000: new disk, initializing [ 2083.433095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.444179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2085.203889] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2089.904494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2089.933606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2089.968992] Lustre: 121529:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2090.007999] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2090.011214] Lustre: Skipped 1 previous similar message [ 2090.051723] Lustre: lustre-MDT0001: new disk, initializing [ 2090.090362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.103448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2090.107433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2091.569263] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2095.288412] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2097.002737] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2100.257896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2100.297143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.451367] Lustre: lustre-OST0000: new disk, initializing [ 2100.454548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2100.506186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.763060] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2106.299138] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2107.539515] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2108.940613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2108.949700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2109.004936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2109.714314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2111.503534] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2111.618319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2114.023438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2114.023556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.027532] LustreError: Skipped 1 previous similar message [ 2114.033880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.037845] Lustre: Skipped 1 previous similar message [ 2118.644185] LustreError: 124483:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2118.707851] Lustre: server umount lustre-OST0000 complete [ 2120.673515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2120.678606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.686544] Lustre: Skipped 1 previous similar message [ 2120.690257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.257530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.258204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.674461] LustreError: 124684:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2126.678910] LustreError: 124684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2126.710880] Lustre: server umount lustre-MDT0000 complete [ 2128.472286] LustreError: 121520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386206 with bad export cookie 10347948080004971207 [ 2128.479078] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.480503] LustreError: 121520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2128.582512] LustreError: 124885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2128.585710] LustreError: 124885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2128.642698] Lustre: server umount lustre-MDT0001 complete [ 2132.182272] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:36:49 (1767386209) [ 2135.040198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.252463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.869315] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2140.598967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2140.642706] LustreError: 125556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.653052] LustreError: 125556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2140.765959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.255878] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2144.508581] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2146.163198] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2150.880978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2150.884891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.891563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.894368] Lustre: Skipped 1 previous similar message [ 2153.161748] LustreError: 126899:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2153.165096] LustreError: 126899:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2153.197406] Lustre: server umount lustre-MDT0000 complete [ 2154.901098] LustreError: 126273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386232 with bad export cookie 10347948080004972075 [ 2154.906369] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.908233] LustreError: 126273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2155.057723] Lustre: server umount lustre-MDT0001 complete [ 2158.765433] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:37:15 (1767386235) [ 2161.134313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2176.608624] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2181.728518] LustreError: 127745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 2207.328333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.722396] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2211.939370] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2212.808799] LustreError: 128409:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2212.811424] LustreError: 128409:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2212.860827] Lustre: server umount lustre-OST0000 complete [ 2216.502162] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:38:13 (1767386293) [ 2218.616486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.792078] LustreError: 129079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.261976] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2223.911280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2224.093440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.095236] Lustre: Skipped 1 previous similar message [ 2225.561136] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2227.539609] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2228.821509] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2231.372115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2233.762794] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2236.276860] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.187624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.191304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.195029] Lustre: Skipped 3 previous similar messages [ 2249.199202] Lustre: Skipped 3 previous similar messages [ 2254.982150] LustreError: 131347:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2254.984274] LustreError: 131347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2255.021783] Lustre: server umount lustre-MDT0000 complete [ 2256.729893] LustreError: 129059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386334 with bad export cookie 10347948080004972726 [ 2256.733229] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.737789] LustreError: 129059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2256.891103] Lustre: server umount lustre-MDT0001 complete [ 2268.908764] Lustre: server umount lustre-OST0000 complete [ 2272.637331] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:39:09 (1767386349) [ 2275.834928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.070397] LustreError: 132422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.111946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.116055] Lustre: Skipped 1 previous similar message [ 2277.885840] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2281.443161] LustreError: 132422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.911373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2283.930462] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2286.306895] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2287.990798] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2291.151956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2293.819744] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2296.434575] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2298.310893] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2300.911474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2301.542880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2303.236269] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2303.356280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2306.021230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.023068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.035768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.041494] Lustre: Skipped 4 previous similar messages [ 2310.195139] LustreError: 135190:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2310.200744] LustreError: 135190:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2310.257744] Lustre: server umount lustre-OST0000 complete [ 2312.673363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2318.190312] Lustre: server umount lustre-MDT0000 complete [ 2319.833135] LustreError: 132403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386397 with bad export cookie 10347948080004973783 [ 2319.835644] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.837518] LustreError: 132403:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2319.951371] Lustre: server umount lustre-MDT0001 complete [ 2323.557767] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:40:00 (1767386400) [ 2326.335914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2341.792578] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2346.912567] LustreError: 136236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 2372.512341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.513949] Lustre: Skipped 2 previous similar messages [ 2374.731709] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2377.276307] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2380.387897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.616226] LustreError: 136263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.702145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2382.395490] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2386.402698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2387.847867] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2389.555032] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2390.739834] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.105615] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2392.179587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2393.464123] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2393.530622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2397.154126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2397.157822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.163632] Lustre: Skipped 3 previous similar messages [ 2397.166279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.169566] Lustre: Skipped 3 previous similar messages [ 2400.331814] LustreError: 139016:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2400.335669] LustreError: 139016:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2400.369473] Lustre: server umount lustre-MDT0000 complete [ 2401.915801] LustreError: 137191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386479 with bad export cookie 10347948080004974791 [ 2401.920827] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2401.921157] LustreError: 137191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2402.061113] Lustre: server umount lustre-MDT0001 complete [ 2414.064753] Lustre: server umount lustre-OST0000 complete [ 2416.815866] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:41:34 (1767386494) [ 2419.935600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2435.424467] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2440.544530] LustreError: 140062:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 2468.582212] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2470.655878] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2473.824565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.031498] LustreError: 140089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.038247] LustreError: 140089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2474.096090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2475.399090] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2478.621837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2480.565919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2482.943239] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2484.667793] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2488.038344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.081583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2488.178792] Lustre: lustre-OST0001: new disk, initializing [ 2488.182799] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2490.660512] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2494.376500] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2494.473412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2494.477916] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2494.526661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2496.190098] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2496.323919] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2498.122932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2498.244754] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2499.563775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2499.564371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.567923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2499.571941] Lustre: Skipped 3 previous similar messages [ 2499.577370] Lustre: Skipped 4 previous similar messages [ 2513.376191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2513.402148] LustreError: 143796:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2513.406199] LustreError: 143796:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2513.434905] Lustre: server umount lustre-OST0000 complete [ 2514.918408] LustreError: 143208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.928077] LustreError: 143208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2521.164705] Lustre: server umount lustre-OST0001 complete [ 2528.989245] Lustre: server umount lustre-MDT0000 complete [ 2530.623484] LustreError: 140070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386608 with bad export cookie 10347948080004975729 [ 2530.626714] LustreError: 140070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2530.627420] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.739337] Lustre: server umount lustre-MDT0001 complete [ 2532.464150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2534.094949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.729722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2537.397653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.646089] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:43:37 (1767386617) [ 2541.371107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2541.998754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2542.500219] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:43:39 (1767386619) [ 2545.646802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.793566] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2545.812862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2545.917887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.921490] Lustre: Skipped 6 previous similar messages [ 2547.652558] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2550.675440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2550.717482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2550.730383] Lustre: 146038:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2552.543867] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2554.951380] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2556.609148] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2559.886968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2560.002234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2562.545460] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2563.056756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2564.788209] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2566.533099] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2566.662562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2568.429290] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2568.502286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2573.285694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2573.286535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.288404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.288412] Lustre: Skipped 8 previous similar messages [ 2573.289706] LustreError: Skipped 5 previous similar messages [ 2573.294071] Lustre: Skipped 5 previous similar messages [ 2575.272968] Lustre: server umount lustre-OST0000 complete [ 2590.127521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2592.091425] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2593.381298] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.384192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.384260] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2593.388699] Lustre: Skipped 1 previous similar message [ 2593.898121] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2595.688081] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2595.792885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2597.578174] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2597.691375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2599.277164] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ee890c4f800.ost_server_uuid 50 [ 2600.023855] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ee890c4f800.ost_server_uuid in FULL state after 0 sec [ 2600.079626] LustreError: 146043:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.6@tcp) returned error from blocking AST (req@00000000eff9fc4f x1853238690378496 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ebc23268/0x8f9b4bf3f6e1d5bd lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.6@tcp remote: 0x5f9d4b458fb6f773 expref: 5 pid: 146044 timeout: 2700 lvb_type: 0 [ 2600.087712] LustreError: lustre-MDT0000: A client on nid 192.168.203.6@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2600.090070] LustreError: 146033:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.6@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ebc23268/0x8f9b4bf3f6e1d5bd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.6@tcp remote: 0x5f9d4b458fb6f773 expref: 6 pid: 146044 timeout: 0 lvb_type: 0 [ 2607.143913] Lustre: server umount lustre-OST0000 complete [ 2615.016396] Lustre: server umount lustre-MDT0000 complete [ 2616.547813] LustreError: 146029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386694 with bad export cookie 10347948080004976912 [ 2616.555075] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.557372] LustreError: 146029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2622.778268] Lustre: server umount lustre-MDT0001 complete [ 2626.460922] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2627.665741] Key type lgssc unregistered [ 2627.794387] LNet: 151521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2627.797296] LNetError: 151521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2627.803429] LNet: Removed LNI 192.168.203.106@tcp [ 2628.106928] Key type .llcrypt unregistered [ 2628.108259] Key type ._llcrypt unregistered [ 2637.904159] Key type ._llcrypt registered [ 2637.905343] Key type .llcrypt registered [ 2637.958987] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2638.495619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.506145] alg: No test for adler32 (adler32-zlib) [ 2639.369108] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2639.447242] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2641.024102] Key type lgssc registered [ 2641.364336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2644.583161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2644.590657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.764197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.377445] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2650.959801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2651.135279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2652.336367] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2654.049117] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2655.439697] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2658.632329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2658.822129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.345765] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2663.013181] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.142736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2675.919738] Lustre: Failing over lustre-MDT0000 [ 2676.122107] LustreError: 155157:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2676.165496] Lustre: server umount lustre-MDT0000 complete [ 2676.196860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.199954] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.202584] Lustre: Skipped 2 previous similar messages [ 2676.207496] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2681.319166] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.328618] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2686.435702] LustreError: 152890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.443288] LustreError: 152890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2687.316774] LustreError: 154511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1767386765 with bad export cookie 9636548987338593931 [ 2687.325339] LustreError: 154511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2690.621921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.667610] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.803829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.835473] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2692.244171] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2692.952298] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2692.956463] Lustre: lustre-MDT0000: Denying connection for new client 7327571a-d570-40a3-aa80-baa20c538cb4 (at 192.168.203.6@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2696.165702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2701.284573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.288408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2701.292313] Lustre: Skipped 1 previous similar message [ 2705.965591] LustreError: 156094:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2705.967342] LustreError: 156094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2705.998798] Lustre: server umount lustre-OST0000 complete [ 2707.097982] LustreError: 156296:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2707.100226] Lustre: 155659:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2707.102549] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2707.108507] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2707.112206] LustreError: 155659:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bb37c638 x1853239294200320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2707.116949] LustreError: 155659:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2707.119618] LustreError: 155659:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2707.122388] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2707.196379] LustreError: 156296:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2707.198024] LustreError: 156296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2707.224793] Lustre: server umount lustre-MDT0000 complete [ 2708.364879] LustreError: 152870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386786 with bad export cookie 9636548987338594288 [ 2708.370597] LustreError: 152200:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c7b8e14 x1853239294201216/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2708.371440] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.512624] LustreError: 156498:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2714.516847] LustreError: 156498:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2714.600740] Lustre: server umount lustre-MDT0001 complete [ 2719.446273] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2720.988670] Key type lgssc unregistered [ 2721.151575] LNet: 157030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.156288] LNetError: 157030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.165652] LNet: Removed LNI 192.168.203.106@tcp [ 2721.617481] Key type .llcrypt unregistered [ 2721.619848] Key type ._llcrypt unregistered [ 2733.199924] Key type ._llcrypt registered [ 2733.201172] Key type .llcrypt registered [ 2733.262245] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2733.678578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2733.684311] alg: No test for adler32 (adler32-zlib) [ 2734.608062] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2734.731791] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2736.328159] Key type lgssc registered [ 2736.955412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.052296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.063733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.262947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.841984] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2747.553228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2747.777921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.624675] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2751.660209] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2753.080545] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2756.031770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2756.212393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.885496] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2761.483964] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2767.475177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2778.085195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.086298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.093866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.100026] Lustre: Skipped 1 previous similar message [ 2778.109101] Lustre: Skipped 1 previous similar message [ 2783.202617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.206838] Lustre: Skipped 1 previous similar message [ 2788.320125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2788.323709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2788.330131] Lustre: Skipped 1 previous similar message [ 2788.354150] LustreError: 160666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2788.409812] Lustre: server umount lustre-OST0000 complete [ 2793.443147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.444282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.450576] Lustre: Skipped 1 previous similar message [ 2796.295099] LustreError: 160869:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2796.297715] LustreError: 160869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2796.327368] Lustre: server umount lustre-MDT0000 complete [ 2797.675932] LustreError: 160567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386875 with bad export cookie 8130660382632886367 [ 2797.679623] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2797.680677] LustreError: 160567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2797.747260] LustreError: 161071:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2797.749664] LustreError: 161071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2797.793633] Lustre: server umount lustre-MDT0001 complete [ 2801.400022] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2802.501180] Key type lgssc unregistered [ 2802.653578] LNet: 161602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2802.656868] LNetError: 161602:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2802.665426] LNet: Removed LNI 192.168.203.106@tcp [ 2803.063154] Key type .llcrypt unregistered [ 2803.064481] Key type ._llcrypt unregistered [ 2816.762903] Key type ._llcrypt registered [ 2816.764352] Key type .llcrypt registered [ 2816.827267] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2817.316741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.366543] alg: No test for adler32 (adler32-zlib) [ 2818.266630] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2818.397251] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2820.008149] Key type lgssc registered [ 2820.596899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.998178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2824.009078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.269296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.021788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2830.910079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2831.058154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2832.812919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2835.363591] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2837.150675] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2840.423797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2840.596916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.215128] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2845.789719] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2851.882910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2861.030204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.031351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.038482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.044753] Lustre: Skipped 1 previous similar message [ 2861.051510] Lustre: Skipped 1 previous similar message [ 2865.075150] LustreError: 165531:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2865.130235] Lustre: server umount lustre-OST0000 complete [ 2867.169145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2867.173132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.180026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.265149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.265727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.964964] LustreError: 165734:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2872.967744] LustreError: 165734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2872.995439] Lustre: server umount lustre-MDT0000 complete [ 2874.721754] LustreError: 163242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386952 with bad export cookie 3924765117549497268 [ 2874.726124] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2874.727443] LustreError: 163242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2874.811682] LustreError: 165936:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2874.813663] LustreError: 165936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2874.856821] Lustre: server umount lustre-MDT0001 complete [ 2879.617767] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2881.082360] Key type lgssc unregistered [ 2881.230713] LNet: 166467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2881.235359] LNetError: 166467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2881.244579] LNet: Removed LNI 192.168.203.106@tcp [ 2881.710991] Key type .llcrypt unregistered [ 2881.713936] Key type ._llcrypt unregistered [ 2892.482732] Key type ._llcrypt registered [ 2892.483749] Key type .llcrypt registered [ 2892.523378] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2892.893000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2892.899808] alg: No test for adler32 (adler32-zlib) [ 2893.740432] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2893.829943] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2895.408163] Key type lgssc registered [ 2896.006157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2901.156455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2901.167102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.324365] Lustre: *** cfs_fail_loc=135, val=0*** [ 2902.363859] LustreError: 167951:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2902.365647] LustreError: 167951:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -2 [ 2902.367751] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2902.367751] [ 2902.370024] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2902.373459] LustreError: 167911:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2902.375343] LustreError: 167911:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -2 [ 2902.377213] LustreError: 167911:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2902.378791] LustreError: 167911:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2902.381191] LustreError: 167911:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2902.411376] Lustre: server umount lustre-MDT0000 complete [ 2902.412647] LustreError: 167911:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2905.681955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.923773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.710552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2911.862550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2912.077643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2913.929165] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2916.665839] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2918.488307] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.443718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.445626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.451043] Lustre: Skipped 1 previous similar message [ 2922.467160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.470869] Lustre: Skipped 2 previous similar messages [ 2925.646330] LustreError: 169668:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2925.651806] LustreError: 169668:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2925.695524] Lustre: server umount lustre-MDT0000 complete [ 2927.536663] LustreError: 168306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387005 with bad export cookie 14989372701038033208 [ 2927.544593] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.546459] LustreError: 168306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2927.652078] LustreError: 169872:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2927.655971] LustreError: 169872:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2927.726477] Lustre: server umount lustre-MDT0001 complete [ 2932.341706] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2933.578948] Key type lgssc unregistered [ 2933.709420] LNet: 170403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.713066] LNetError: 170403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.721418] LNet: Removed LNI 192.168.203.106@tcp [ 2934.097138] Key type .llcrypt unregistered [ 2934.098992] Key type ._llcrypt unregistered [ 2943.638888] Key type ._llcrypt registered [ 2943.640726] Key type .llcrypt registered [ 2943.701078] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2943.870321] Key type .llcrypt unregistered [ 2943.872691] Key type ._llcrypt unregistered [ 2952.376239] Key type ._llcrypt registered [ 2952.378439] Key type .llcrypt registered [ 2952.440192] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2952.970278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2952.977834] alg: No test for adler32 (adler32-zlib) [ 2953.877049] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2953.978260] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2955.576197] Key type lgssc registered [ 2956.136280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2960.023234] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2960.031476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2975.520519] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2980.640268] LustreError: 172473:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 3007.264251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.074543] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3012.523031] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3015.700060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.932171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.961510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3017.711315] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3021.285775] LustreError: 173419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.297207] LustreError: 173419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3021.965795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3022.156198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.023977] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3026.654132] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3028.410531] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3029.788402] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3033.580137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.582668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.589076] Lustre: Skipped 1 previous similar message [ 3033.595402] Lustre: Skipped 1 previous similar message [ 3038.698377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.702120] Lustre: Skipped 1 previous similar message [ 3038.773171] LustreError: 175005:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3038.837526] Lustre: server umount lustre-OST0000 complete [ 3042.785706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.790695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.799631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.812500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.784099] LustreError: 175207:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3046.785973] LustreError: 175207:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3046.822231] Lustre: server umount lustre-MDT0000 complete [ 3048.537285] LustreError: 174802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387126 with bad export cookie 17343455940832075859 [ 3048.544373] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.544521] LustreError: 174802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3048.646566] LustreError: 175408:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3048.650558] LustreError: 175408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3048.709166] Lustre: server umount lustre-MDT0001 complete [ 3052.875553] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3054.103887] Key type lgssc unregistered [ 3054.234590] LNet: 175939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.236957] LNetError: 175939:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3054.248519] LNet: Removed LNI 192.168.203.106@tcp [ 3054.613459] Key type .llcrypt unregistered [ 3054.615313] Key type ._llcrypt unregistered [ 3066.345615] Key type ._llcrypt registered [ 3066.347523] Key type .llcrypt registered [ 3066.416323] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3066.811286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.826064] alg: No test for adler32 (adler32-zlib) [ 3067.712778] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3067.795475] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3069.376185] Key type lgssc registered [ 3069.982039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3073.790192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3073.801278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.006738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.808847] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3081.075132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.281909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.818160] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3085.313667] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3086.894393] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3090.121567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3090.331764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.178311] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3095.776315] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3099.573803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3102.431046] Lustre: Failing over lustre-MDT0000 [ 3102.534920] LustreError: 179607:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3102.582147] Lustre: server umount lustre-MDT0000 complete [ 3104.742681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.746128] LustreError: 177304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.751205] Lustre: Skipped 2 previous similar messages [ 3104.761232] LustreError: 177304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3106.135639] LustreError: 177304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.858602] LustreError: 177310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.869367] LustreError: 177310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3114.977889] LustreError: 177310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.982090] LustreError: 177310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3116.652347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.695207] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.826993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.872578] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3118.600605] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3120.813768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3121.485156] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3122.160938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3122.172288] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3122.204437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3127.358508] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3137.509646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.511401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3137.512164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3137.518643] Lustre: Skipped 1 previous similar message [ 3139.059137] LustreError: 180945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3139.064102] LustreError: 180945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3139.117038] Lustre: server umount lustre-OST0000 complete [ 3142.628221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.631192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.632287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3142.632297] Lustre: Skipped 1 previous similar message [ 3142.634036] LustreError: Skipped 1 previous similar message [ 3142.651258] Lustre: Skipped 1 previous similar message [ 3147.024737] LustreError: 181146:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3147.028897] LustreError: 181146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3147.071742] Lustre: server umount lustre-MDT0000 complete [ 3147.746685] LustreError: 177309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.756911] LustreError: 177309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3148.901298] LustreError: 177292:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387226 with bad export cookie 16632389631098596660 [ 3148.907135] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.911963] LustreError: 177292:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3149.026897] LustreError: 181347:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3149.031686] LustreError: 181347:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3149.112282] Lustre: server umount lustre-MDT0001 complete [ 3153.382382] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3154.402342] Key type lgssc unregistered [ 3154.563540] LNet: 181878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3154.566324] LNetError: 181878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3154.578610] LNet: Removed LNI 192.168.203.106@tcp [ 3154.999592] Key type .llcrypt unregistered [ 3155.001249] Key type ._llcrypt unregistered [ 3165.434250] Key type ._llcrypt registered [ 3165.435966] Key type .llcrypt registered [ 3165.504727] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3166.062727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3166.092211] alg: No test for adler32 (adler32-zlib) [ 3166.974388] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3167.096873] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3168.688233] Key type lgssc registered [ 3169.249038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3173.002218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3173.013674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.245452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.979581] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3180.182670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3180.381890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.132706] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3184.853934] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3186.616717] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3189.899470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3190.082736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.454695] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3195.096624] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3199.345251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3200.944080] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3207.815596] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3216.212896] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3225.065942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.066557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.069420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.082021] Lustre: Skipped 1 previous similar message [ 3228.852236] LustreError: 185662:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3228.913868] Lustre: server umount lustre-OST0000 complete [ 3231.713449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3231.718479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.727266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.731188] Lustre: Skipped 1 previous similar message [ 3235.297424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.298248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.814699] LustreError: 185868:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3236.818654] LustreError: 185868:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3236.859783] Lustre: server umount lustre-MDT0000 complete [ 3238.660082] LustreError: 183230:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387316 with bad export cookie 13254087710515442180 [ 3238.667190] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.667318] LustreError: 183230:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3238.775475] LustreError: 186070:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3238.777933] LustreError: 186070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3238.842316] Lustre: server umount lustre-MDT0001 complete [ 3243.469821] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3244.802468] Key type lgssc unregistered [ 3244.955419] LNet: 186601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.959719] LNetError: 186601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.972569] LNet: Removed LNI 192.168.203.106@tcp [ 3245.321466] Key type .llcrypt unregistered [ 3245.322401] Key type ._llcrypt unregistered [ 3257.274630] Key type ._llcrypt registered [ 3257.276540] Key type .llcrypt registered [ 3257.336689] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3257.742085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3257.748976] alg: No test for adler32 (adler32-zlib) [ 3258.637410] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3258.738041] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3260.312184] Key type lgssc registered [ 3260.885325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3264.305325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3264.310575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.482842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.032002] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3271.005133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3271.196731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.119355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3275.775793] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.613390] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3280.657761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3280.830571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.399583] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3286.053357] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3291.122423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3293.869705] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3298.457839] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3307.276159] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3317.031334] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3325.924383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.924632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3325.929235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3325.933273] Lustre: Skipped 1 previous similar message [ 3330.483180] LustreError: 191947:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3330.537472] Lustre: server umount lustre-OST0000 complete [ 3332.577329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3332.581984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.589018] Lustre: Skipped 1 previous similar message [ 3332.592254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.161986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.163361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.444895] LustreError: 192149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3338.449399] LustreError: 192149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3338.481891] Lustre: server umount lustre-MDT0000 complete [ 3340.207967] LustreError: 187954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387418 with bad export cookie 11515288410619175796 [ 3340.213280] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3340.217979] LustreError: 187954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3340.313569] LustreError: 192350:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3340.315957] LustreError: 192350:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3340.363842] Lustre: server umount lustre-MDT0001 complete [ 3344.956508] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3346.292858] Key type lgssc unregistered [ 3346.479789] LNet: 192881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3346.484955] LNetError: 192881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3346.495865] LNet: Removed LNI 192.168.203.106@tcp [ 3346.975376] Key type .llcrypt unregistered [ 3346.977806] Key type ._llcrypt unregistered [ 3357.542358] Key type ._llcrypt registered [ 3357.544057] Key type .llcrypt registered [ 3357.616807] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3358.030340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3358.076638] alg: No test for adler32 (adler32-zlib) [ 3358.977649] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3359.067948] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3360.664158] Key type lgssc registered [ 3361.295568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3364.590120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3364.601750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.838895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.374141] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3370.671394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3370.796424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.421354] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3374.765447] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3376.470720] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3379.604788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3379.804585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.513769] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3384.902385] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3391.087304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3401.703860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.708145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.713147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.716042] LustreError: Skipped 1 previous similar message [ 3401.719195] Lustre: Skipped 1 previous similar message [ 3406.005170] LustreError: 196670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3406.069256] Lustre: server umount lustre-OST0000 complete [ 3411.940319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.940583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3411.942597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.947653] Lustre: Skipped 2 previous similar messages [ 3417.058509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3417.061738] Lustre: Skipped 2 previous similar messages [ 3422.176675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3422.179884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.186737] Lustre: Skipped 1 previous similar message [ 3422.241448] LustreError: 196872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3422.244740] LustreError: 196872:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3422.277508] Lustre: server umount lustre-MDT0000 complete [ 3424.069389] LustreError: 194231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387501 with bad export cookie 9816646489980675320 [ 3424.076591] LustreError: 194231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3424.077982] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.185399] LustreError: 197075:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3424.189499] LustreError: 197075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3424.268501] Lustre: server umount lustre-MDT0001 complete [ 3429.050598] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3430.444820] Key type lgssc unregistered [ 3430.617782] LNet: 197606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3430.622557] LNetError: 197606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3430.635702] LNet: Removed LNI 192.168.203.106@tcp [ 3431.082602] Key type .llcrypt unregistered [ 3431.084866] Key type ._llcrypt unregistered [ 3439.966163] Key type ._llcrypt registered [ 3439.967476] Key type .llcrypt registered [ 3440.027229] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3440.368911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3440.409815] alg: No test for adler32 (adler32-zlib) [ 3441.311574] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3441.421584] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3443.024205] Key type lgssc registered [ 3443.666832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.599474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.610992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.857844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.419560] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3454.279829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3454.475170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3456.274962] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3458.412656] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3460.256669] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3463.542251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3463.720825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3466.072922] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3468.518993] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3473.969663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3479.013724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.016705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.021620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3479.025761] Lustre: Skipped 1 previous similar message [ 3480.628226] LustreError: 201226:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3480.691244] Lustre: server umount lustre-OST0000 complete [ 3484.132507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.133698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.140065] Lustre: Skipped 1 previous similar message [ 3484.143449] Lustre: Skipped 1 previous similar message [ 3484.558178] LustreError: 201428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3484.562184] LustreError: 201428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3484.602432] Lustre: server umount lustre-MDT0000 complete [ 3486.314261] LustreError: 200994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387564 with bad export cookie 7274000301428763175 [ 3486.321061] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.322476] LustreError: 200994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3486.433496] LustreError: 201629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3486.437261] LustreError: 201629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3486.531094] Lustre: server umount lustre-MDT0001 complete [ 3491.416439] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3492.588392] Key type lgssc unregistered [ 3492.748659] LNet: 202160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3492.753898] LNetError: 202160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3492.762572] LNet: Removed LNI 192.168.203.106@tcp [ 3493.153358] Key type .llcrypt unregistered [ 3493.154378] Key type ._llcrypt unregistered [ 3502.315810] Key type ._llcrypt registered [ 3502.317744] Key type .llcrypt registered [ 3502.374945] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3502.720227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3502.736905] alg: No test for adler32 (adler32-zlib) [ 3503.600321] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3503.706991] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3505.296163] Key type lgssc registered [ 3505.841900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.718833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.732083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.944920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.375480] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3516.086340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3516.256151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3518.022673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3520.800795] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.672642] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3525.969831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3526.118619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.578860] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3531.131296] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.342250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3540.019067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3540.059755] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3540.187685] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3542.745874] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3545.322362] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3546.035488] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3556.696311] Lustre: Permanently deactivating lustre-OST0001 [ 3556.706533] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3557.052626] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3567.154404] Lustre: Permanently reactivating lustre-OST0001 [ 3567.165160] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3567.169801] Lustre: Skipped 2 previous similar messages [ 3573.782770] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.791282] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3573.796171] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3573.805161] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3573.826545] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3578.851910] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.853928] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.857521] LustreError: Skipped 1 previous similar message [ 3578.864691] Lustre: Skipped 1 previous similar message [ 3578.866418] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3578.877831] Lustre: Skipped 1 previous similar message [ 3581.877153] LustreError: 206832:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3581.901974] Lustre: server umount lustre-OST0001 complete [ 3583.972851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.973197] LustreError: 204981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.981710] Lustre: Skipped 1 previous similar message [ 3583.997105] LustreError: 204981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3584.006861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.577518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.582171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.590756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.094977] LustreError: 204983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.105327] LustreError: 204983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3589.813141] LustreError: 207034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3589.820303] LustreError: 207034:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3589.877232] Lustre: server umount lustre-OST0000 complete [ 3593.185283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3593.189805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.198106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.201793] Lustre: Skipped 2 previous similar messages [ 3597.835941] LustreError: 207248:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3597.838022] LustreError: 207248:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3597.865218] Lustre: server umount lustre-MDT0000 complete [ 3599.329555] LustreError: 203385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.334818] LustreError: 203385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3599.340391] LustreError: 205893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387677 with bad export cookie 7903640210444344934 [ 3599.343032] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.343527] LustreError: 205893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3599.470745] Lustre: server umount lustre-MDT0001 complete [ 3603.357481] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3604.509526] Key type lgssc unregistered [ 3604.654434] LNet: 207980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3604.656879] LNetError: 207980:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3604.667420] LNet: Removed LNI 192.168.203.106@tcp [ 3605.034716] Key type .llcrypt unregistered [ 3605.035686] Key type ._llcrypt unregistered [ 3617.764754] Key type ._llcrypt registered [ 3617.766021] Key type .llcrypt registered [ 3617.807798] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3625.439653] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3625.710558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3625.719853] alg: No test for adler32 (adler32-zlib) [ 3626.641659] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3626.775764] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3628.384159] Key type lgssc registered [ 3629.028300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3633.523123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3636.703769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.710068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3642.775656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3646.043188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3646.080952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3646.090871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.198751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3647.214396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3647.277813] Lustre: lustre-MDT0000: new disk, initializing [ 3647.319473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.331410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3649.200871] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3654.442498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.473324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3654.513077] Lustre: 211899:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3654.536909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3654.538549] Lustre: Skipped 1 previous similar message [ 3654.573283] Lustre: lustre-MDT0001: new disk, initializing [ 3654.609163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3654.622441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3654.625602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3656.281654] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3659.713932] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3660.995552] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3663.694401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3663.723918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3663.850919] Lustre: lustre-OST0000: new disk, initializing [ 3663.852870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3663.891927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.345568] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3670.385391] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3671.050808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3671.056800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3671.115577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3672.191114] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3672.272861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3673.827703] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3673.927729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3676.130993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.133683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.136380] LustreError: Skipped 1 previous similar message [ 3676.148768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.154349] Lustre: Skipped 1 previous similar message [ 3680.881156] LustreError: 214835:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3680.938222] Lustre: server umount lustre-OST0000 complete [ 3685.345521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3685.350435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.358068] Lustre: Skipped 1 previous similar message [ 3685.361504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.369449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.370399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.490105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3691.494649] Lustre: Skipped 2 previous similar messages [ 3696.610256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.614410] Lustre: Skipped 1 previous similar message [ 3697.120146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3697.179657] LustreError: 215036:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3697.184372] LustreError: 215036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3697.218812] Lustre: server umount lustre-MDT0000 complete [ 3699.009118] LustreError: 211891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387776 with bad export cookie 16637795716026619390 [ 3699.015855] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.016967] LustreError: 211891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3699.110882] LustreError: 215238:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3699.114926] LustreError: 215238:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3699.174549] Lustre: server umount lustre-MDT0001 complete [ 3702.601382] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 16:02:59 (1767387779) [ 3705.857139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.110551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.896934] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3711.460226] LustreError: 215909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.470469] LustreError: 215909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3712.019542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3712.225232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3714.098207] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3716.645235] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3718.343489] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3721.673839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3721.863547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.487697] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3727.024291] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3733.842952] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3735.437952] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3742.242901] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3752.160672] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3767.821707] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3767.826868] Lustre: Skipped 1 previous similar message [ 3791.242135] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3791.244838] Lustre: Skipped 2 previous similar messages [ 3828.939827] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3828.947561] Lustre: Skipped 3 previous similar messages [ 3893.802741] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3893.807750] Lustre: Skipped 8 previous similar messages [ 3900.389169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.389629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3900.391720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3900.394135] Lustre: Skipped 1 previous similar message [ 3901.747329] LustreError: 219391:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3901.751249] LustreError: 219391:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3901.800880] Lustre: server umount lustre-OST0000 complete [ 3905.507065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.653013] LustreError: 219596:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3909.656407] LustreError: 219596:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3909.689286] Lustre: server umount lustre-MDT0000 complete [ 3910.625618] LustreError: 215909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.634943] LustreError: 215909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3911.338548] LustreError: 219122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387989 with bad export cookie 16637795716026620258 [ 3911.343321] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3911.343420] LustreError: 219122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3911.480374] Lustre: server umount lustre-MDT0001 complete [ 3915.657761] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3917.254967] Key type lgssc unregistered [ 3917.437811] LNet: 220329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3917.442804] LNetError: 220329:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3917.453583] LNet: Removed LNI 192.168.203.106@tcp [ 3917.905797] Key type .llcrypt unregistered [ 3917.907945] Key type ._llcrypt unregistered [ 3929.692979] Key type ._llcrypt registered [ 3929.694138] Key type .llcrypt registered [ 3929.736774] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3930.136396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3930.153565] alg: No test for adler32 (adler32-zlib) [ 3931.065732] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3931.187528] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3932.800217] Key type lgssc registered [ 3933.404729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3937.470642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3937.482834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.742071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.644243] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3944.963543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3945.186707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.090024] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3949.729578] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3951.568376] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3954.833813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3954.975806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.539639] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3960.219328] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3979.747451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3979.750583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.753834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3979.756091] Lustre: Skipped 1 previous similar message [ 3980.772909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.777085] Lustre: Skipped 1 previous similar message [ 3983.796367] LustreError: 224104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3983.856889] Lustre: server umount lustre-OST0000 complete [ 3985.891549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.892734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.900510] Lustre: Skipped 1 previous similar message [ 3992.034571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3992.038611] Lustre: Skipped 4 previous similar messages [ 3997.154062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.158025] Lustre: Skipped 1 previous similar message [ 4000.224193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4000.283747] LustreError: 224309:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4000.288889] LustreError: 224309:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4000.322578] Lustre: server umount lustre-MDT0000 complete [ 4002.096923] LustreError: 222421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388079 with bad export cookie 10298367686148571691 [ 4002.104074] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4002.104473] LustreError: 222421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4002.196667] LustreError: 224513:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4002.199058] LustreError: 224513:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4002.248807] Lustre: server umount lustre-MDT0001 complete [ 4007.189892] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4008.757700] Key type lgssc unregistered [ 4008.930696] LNet: 225045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.935813] LNetError: 225045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4008.944697] LNet: Removed LNI 192.168.203.106@tcp [ 4009.377885] Key type .llcrypt unregistered [ 4009.380157] Key type ._llcrypt unregistered [ 4020.880553] Key type ._llcrypt registered [ 4020.881852] Key type .llcrypt registered [ 4020.950282] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4021.411640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4021.475768] alg: No test for adler32 (adler32-zlib) [ 4022.364330] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4022.469639] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4024.056162] Key type lgssc registered [ 4024.694287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4290.614170] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4292.024432] Key type lgssc unregistered [ 4292.180538] LNet: 227008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4292.185192] LNetError: 227008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4292.194562] LNet: Removed LNI 192.168.203.106@tcp [ 4292.675687] Key type .llcrypt unregistered [ 4292.677439] Key type ._llcrypt unregistered [ 4303.792131] Key type ._llcrypt registered [ 4303.793925] Key type .llcrypt registered [ 4303.834108] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4311.722695] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4312.020327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4312.027831] alg: No test for adler32 (adler32-zlib) [ 4312.881437] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4312.978816] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4314.576091] Key type lgssc registered [ 4315.090039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4319.155996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4322.157498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4325.069609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4328.397989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4329.284553] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4204 sec ========== 16:13:26 (1767388406) [ 4330.053883] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:13:27 (1767388407) === [ 4331.416623] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:13:28 (1767388408) === [ 4349.716390] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4351.104419] Key type lgssc unregistered [ 4351.259441] LNet: 232169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4351.264255] LNetError: 232169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4351.272500] LNet: Removed LNI 192.168.203.106@tcp [ 4351.676828] Key type .llcrypt unregistered [ 4351.680043] Key type ._llcrypt unregistered