[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 406424604 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002251] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006547] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007009] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009083] LSM: Security Framework initializing [ 0.010027] Yama: becoming mindful. [ 0.011019] SELinux: Initializing. [ 0.011805] *** VALIDATE selinux *** [ 0.018648] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022515] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024063] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026032] *** VALIDATE tmpfs *** [ 0.027139] *** VALIDATE proc *** [ 0.028040] *** VALIDATE cgroup *** [ 0.028790] *** VALIDATE cgroup2 *** [ 0.029164] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030106] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032016] Spectre V2 : User space: Vulnerable [ 0.033003] Speculative Store Bypass: Vulnerable [ 0.035645] debug: unmapping init [mem 0xffffffffb6c59000-0xffffffffb6c60fff] [ 0.037119] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038445] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039010] ... version: 2 [ 0.039893] ... bit width: 48 [ 0.040006] ... generic registers: 4 [ 0.040945] ... value mask: 0000ffffffffffff [ 0.041006] ... max period: 00007fffffffffff [ 0.042006] ... fixed-purpose events: 3 [ 0.043006] ... event mask: 000000070000000f [ 0.044199] rcu: Hierarchical SRCU implementation. [ 0.046094] smp: Bringing up secondary CPUs ... [ 0.047358] x86: Booting SMP configuration: [ 0.048014] .... node #0, CPUs: #1 #2 #3 [ 0.050539] smp: Brought up 1 node, 4 CPUs [ 0.051862] smpboot: Max logical packages: 1 [ 0.052007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146324] node 0 deferred pages initialised in 93ms [ 0.149012] devtmpfs: initialized [ 0.150220] x86/mm: Memory block size: 128MB [ 0.152344] gcov: version magic: 0x41383552 [ 0.153618] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155256] pinctrl core: initialized pinctrl subsystem [ 0.156127] [ 0.156837] ************************************************************* [ 0.157025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158010] ** ** [ 0.159009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160011] ** ** [ 0.161007] ** This means that this kernel is built to expose internal ** [ 0.162006] ** IOMMU data structures, which may compromise security on ** [ 0.163006] ** your system. ** [ 0.164008] ** ** [ 0.165007] ** If you see this message and you are not debugging the ** [ 0.166010] ** kernel, report this immediately to your vendor! ** [ 0.167010] ** ** [ 0.168007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169006] ************************************************************* [ 0.170444] NET: Registered protocol family 16 [ 0.171299] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174329] cpuidle: using governor menu [ 0.175359] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177368] PCI: Using configuration type 1 for base access [ 0.178087] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189120] cryptd: max_cpu_qlen set to 1000 [ 0.190158] ACPI: Added _OSI(Module Device) [ 0.192060] ACPI: Added _OSI(Processor Device) [ 0.193009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200321] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207611] ACPI: Interpreter enabled [ 0.209062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210009] ACPI: Using IOAPIC for interrupt routing [ 0.212119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217419] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225741] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.232035] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234952] acpiphp: Slot [2] registered [ 0.236110] acpiphp: Slot [5] registered [ 0.238102] acpiphp: Slot [6] registered [ 0.239072] acpiphp: Slot [7] registered [ 0.240035] acpiphp: Slot [8] registered [ 0.241059] acpiphp: Slot [9] registered [ 0.243091] acpiphp: Slot [10] registered [ 0.244067] acpiphp: Slot [3] registered [ 0.245045] acpiphp: Slot [4] registered [ 0.245978] acpiphp: Slot [11] registered [ 0.247057] acpiphp: Slot [12] registered [ 0.248061] acpiphp: Slot [13] registered [ 0.249000] acpiphp: Slot [14] registered [ 0.249953] acpiphp: Slot [15] registered [ 0.251045] acpiphp: Slot [16] registered [ 0.252036] acpiphp: Slot [17] registered [ 0.252922] acpiphp: Slot [18] registered [ 0.254055] acpiphp: Slot [19] registered [ 0.255033] acpiphp: Slot [20] registered [ 0.255958] acpiphp: Slot [21] registered [ 0.256054] acpiphp: Slot [22] registered [ 0.256979] acpiphp: Slot [23] registered [ 0.258045] acpiphp: Slot [24] registered [ 0.258995] acpiphp: Slot [25] registered [ 0.260044] acpiphp: Slot [26] registered [ 0.261053] acpiphp: Slot [27] registered [ 0.261986] acpiphp: Slot [28] registered [ 0.263060] acpiphp: Slot [29] registered [ 0.263971] acpiphp: Slot [30] registered [ 0.265044] acpiphp: Slot [31] registered [ 0.266033] PCI host bridge to bus 0000:00 [ 0.266979] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.271007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.275022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.276111] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277751] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279791] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.288008] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.292493] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.294007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.296008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.297006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.299388] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301565] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.303020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.305474] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.308009] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.318012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.321008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.325115] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.330011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.335010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.347010] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.355416] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.359008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.363007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.375011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.383145] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.387010] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.393012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.404010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.412948] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.419012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.426014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.436013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.446438] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.452012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.457009] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.468013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.474170] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.480011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.485009] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.498014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.508508] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.510225] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.513259] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.514217] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.516106] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.520047] iommu: Default domain type: Passthrough [ 0.521304] SCSI subsystem initialized [ 0.522079] ACPI: bus type USB registered [ 0.524066] usbcore: registered new interface driver usbfs [ 0.525043] usbcore: registered new interface driver hub [ 0.527051] usbcore: registered new device driver usb [ 0.529139] pps_core: LinuxPPS API ver. 1 registered [ 0.531009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.534033] PTP clock support registered [ 0.535070] EDAC MC: Ver: 3.0.0 [ 0.536356] PCI: Using ACPI for IRQ routing [ 0.537575] NetLabel: Initializing [ 0.538006] NetLabel: domain hash size = 128 [ 0.538966] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541039] NetLabel: unlabeled traffic allowed by default [ 0.542076] vgaarb: loaded [ 0.543100] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.544006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.549229] clocksource: Switched to clocksource kvm-clock [ 0.645470] VFS: Disk quotas dquot_6.6.0 [ 0.646551] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.648013] *** VALIDATE ramfs *** [ 0.648712] *** VALIDATE hugetlbfs *** [ 0.649579] pnp: PnP ACPI init [ 0.650980] pnp: PnP ACPI: found 6 devices [ 0.662860] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.664697] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.665881] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.667111] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.668475] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.669981] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.671634] NET: Registered protocol family 2 [ 0.673261] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.676297] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.678423] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.683632] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.685824] TCP: Hash tables configured (established 65536 bind 65536) [ 0.687695] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.691027] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.694425] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.697867] NET: Registered protocol family 1 [ 0.699767] RPC: Registered named UNIX socket transport module. [ 0.701162] RPC: Registered udp transport module. [ 0.702183] RPC: Registered tcp transport module. [ 0.703244] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.704584] NET: Registered protocol family 44 [ 0.705637] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.706993] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.708325] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.709729] PCI: CLS 0 bytes, default 64 [ 0.711394] Unpacking initramfs... [ 1.947665] debug: unmapping init [mem 0xffff9a8f3cc54000-0xffff9a8f3ffbffff] [ 1.950252] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.951684] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.953568] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.406705] Initialise system trusted keyrings [ 2.407812] Key type blacklist registered [ 2.408971] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.416231] zbud: loaded [ 2.418909] *** VALIDATE nfs *** [ 2.420152] *** VALIDATE nfs4 *** [ 2.421579] pstore: using deflate compression [ 2.423733] Platform Keyring initialized [ 2.492262] NET: Registered protocol family 38 [ 2.493716] Key type asymmetric registered [ 2.494803] Asymmetric key parser 'x509' registered [ 2.495917] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.497813] io scheduler mq-deadline registered [ 2.498856] io scheduler kyber registered [ 2.499851] io scheduler bfq registered [ 2.501101] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.502973] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.504742] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.506407] ACPI: Power Button [PWRF] [ 2.509727] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.515546] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.526315] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.531106] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.540726] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.566160] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.592085] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.596168] Non-volatile memory driver v1.3 [ 2.597144] Linux agpgart interface v0.103 [ 2.618816] virtio_blk virtio1: [vda] 150424 512-byte logical blocks (77.0 MB/73.4 MiB) [ 2.620963] vda: detected capacity change from 0 to 77017088 [ 2.632849] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.635792] vdb: detected capacity change from 0 to 1073741824 [ 2.648886] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.651745] vdc: detected capacity change from 0 to 2621440000 [ 2.664136] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.667019] vdd: detected capacity change from 0 to 2621440000 [ 2.678600] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.681542] vde: detected capacity change from 0 to 4294967296 [ 2.692910] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.695152] vdf: detected capacity change from 0 to 4294967296 [ 2.699537] libphy: Fixed MDIO Bus: probed [ 2.702746] usbcore: registered new interface driver usbserial_generic [ 2.704497] usbserial: USB Serial support registered for generic [ 2.705790] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.708442] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.709485] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.711142] mousedev: PS/2 mouse device common for all mice [ 2.713836] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.716495] rtc_cmos 00:05: RTC can wake from S4 [ 2.719522] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.721750] rtc_cmos 00:05: registered as rtc0 [ 2.724780] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.725801] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.732057] intel_pstate: CPU model not supported [ 2.734883] hid: raw HID events driver (C) Jiri Kosina [ 2.736710] usbcore: registered new interface driver usbhid [ 2.738522] usbhid: USB HID core driver [ 2.739869] drop_monitor: Initializing network drop monitor service [ 2.742071] Initializing XFRM netlink socket [ 2.743317] NET: Registered protocol family 10 [ 2.745109] Segment Routing with IPv6 [ 2.745973] NET: Registered protocol family 17 [ 2.747209] mpls_gso: MPLS GSO support [ 2.753688] RAS: Correctable Errors collector initialized. [ 2.756191] AVX version of gcm_enc/dec engaged. [ 2.757846] AES CTR mode by8 optimization enabled [ 2.805226] sched_clock: Marking stable (2805199621, 0)->(3518163562, -712963941) [ 2.807454] registered taskstats version 1 [ 2.809397] Loading compiled-in X.509 certificates [ 2.811725] zswap: loaded using pool lzo/zbud [ 2.829041] Key type big_key registered [ 2.837476] Key type encrypted registered [ 2.838733] ima: No TPM chip found, activating TPM-bypass! [ 2.840423] ima: Allocated hash algorithm: sha1 [ 2.841496] ima: No architecture policies found [ 2.842616] evm: Initialising EVM extended attributes: [ 2.843780] evm: security.selinux [ 2.844526] evm: security.ima [ 2.845133] evm: security.capability [ 2.845878] evm: HMAC attrs: 0x1 [ 2.847469] rtc_cmos 00:05: setting system clock to 2026-09-05 02:16:10 UTC (1788574570) [ 2.852643] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 2.854824] debug: unmapping init [mem 0xffffffffb6982000-0xffffffffb6c58fff] [ 2.861051] Write protecting the kernel read-only data: 28672k [ 2.864339] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 2.866968] debug: unmapping init [mem 0xffffffffb5914000-0xffffffffb59fffff] [ 2.892059] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.897713] systemd[1]: Detected virtualization kvm. [ 2.898808] systemd[1]: Detected architecture x86-64. [ 2.900031] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.921084] systemd[1]: No hostname configured. [ 2.922115] systemd[1]: Set hostname to . [ 2.923333] random: systemd: uninitialized urandom read (16 bytes read) [ 2.925081] systemd[1]: Initializing machine ID from random generator. [ 2.960523] random: ln: uninitialized urandom read (6 bytes read) [ 3.038920] random: systemd: uninitialized urandom read (16 bytes read) [ 3.040855] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.043280] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.045830] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.536660] device-mapper: uevent: version 1.0.3 [ 3.538197] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 4.147996] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.176259] virtio_net virtio0 ens2: renamed from eth0 [ 4.176279] scsi host0: ata_piix [ 4.190708] scsi host1: ata_piix [ 4.192995] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.196046] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.503216] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.354218] random: crng init done [ 9.355993] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.537595] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.467063] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.620794] SELinux: Disabled at runtime. [ 10.675334] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.680768] systemd[1]: Detected virtualization kvm. [ 10.681831] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.001177] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.003230] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.008092] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.010775] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.012572] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.016753] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.021726] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Disp[ 11.059088] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS atch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 11.381234] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.629086] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.671357] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.768597] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.774716] EDAC sbridge: Ver: 1.1.2 [ 12.770860] Key type dns_resolver registered [ 13.046200] NFS: Registering the id_resolver key type [ 13.047501] Key type id_resolver registered [ 13.048387] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg220-server login: [ 34.029956] libcfs: loading out-of-tree module taints kernel. [ 34.047327] Key type ._llcrypt registered [ 34.048577] Key type .llcrypt registered [ 34.083411] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 41.243766] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 41.792359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 41.797866] alg: No test for adler32 (adler32-zlib) [ 42.753935] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 43.026116] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 44.631106] Key type lgssc registered [ 45.166343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 50.955659] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 64.448749] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 68.763897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 68.772320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 69.873131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 69.884732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 69.926958] Lustre: lustre-MDT0000: new disk, initializing [ 69.950275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 69.956608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 71.276701] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 76.286621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.333217] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 76.347591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 76.349753] Lustre: Skipped 1 previous similar message [ 76.388081] Lustre: lustre-MDT0001: new disk, initializing [ 76.410246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 76.419250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 76.424132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 77.807586] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 80.067020] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 83.253043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 83.344566] Lustre: lustre-OST0000: new disk, initializing [ 83.346088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 83.347960] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 83.363526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.292346] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.533689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 87.538449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 87.566217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 90.779029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 90.835373] Lustre: lustre-OST0001: new disk, initializing [ 90.837174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 90.839424] Lustre: 9496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 90.857285] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 92.837657] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.930647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 93.936736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 93.952380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 98.651019] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 102.211240] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 104.264799] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing check_logdir /tmp/testlogs/ [ 105.709202] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing yml_node [ 107.247864] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 108.175322] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 109.035599] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 109.543443] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 22:17:57 EDT 2026 [ 114.855341] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 115.281312] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 127.967604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 127.969969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 127.973665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 130.015644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 130.015945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 130.017944] Lustre: Skipped 1 previous similar message [ 130.021210] Lustre: Skipped 1 previous similar message [ 135.135893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 135.139572] Lustre: Skipped 3 previous similar messages [ 140.255599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 140.258204] Lustre: Skipped 3 previous similar messages [ 141.279145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 141.341897] Lustre: server umount lustre-MDT0000 complete [ 142.564648] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574710 with bad export cookie 17478071620634502786 [ 142.565987] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.570090] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 145.375671] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 145.375904] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 145.379640] Lustre: Skipped 2 previous similar messages [ 145.384475] Lustre: Skipped 1 previous similar message [ 148.703205] Lustre: server umount lustre-MDT0001 complete [ 164.319094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 164.350263] Lustre: server umount lustre-OST0000 complete [ 171.731746] Lustre: server umount lustre-OST0001 complete [ 173.810362] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 176.916744] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 193.548752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.631501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.641143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.676570] Lustre: lustre-MDT0000: new disk, initializing [ 193.699923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.706511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 195.143393] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 199.381834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.430894] Lustre: 15039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.546385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.008382] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 204.909834] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 206.947502] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 209.111597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.216643] Lustre: lustre-OST0000: new disk, initializing [ 209.218320] Lustre: Skipped 1 previous similar message [ 209.220189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.221965] Lustre: Skipped 2 previous similar messages [ 209.224107] Lustre: 17005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.250063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.252561] Lustre: Skipped 1 previous similar message [ 211.004828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.008084] Lustre: Skipped 1 previous similar message [ 211.010348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.046724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.252591] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 214.833408] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 216.156162] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.225776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 217.507845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 217.576357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 221.151900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 221.154118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.158820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 221.161010] Lustre: Skipped 3 previous similar messages [ 224.338963] Lustre: server umount lustre-OST0000 complete [ 226.273921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 226.277063] Lustre: Skipped 2 previous similar messages [ 231.707638] Lustre: server umount lustre-MDT0000 complete [ 232.972345] LustreError: 15029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574800 with bad export cookie 17478071620634504305 [ 232.975297] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 232.976924] LustreError: 15029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 233.093287] Lustre: server umount lustre-MDT0001 complete [ 234.306448] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:20:01 (1788574801) [ 236.578929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.748793] LustreError: 19268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.754755] LustreError: 19268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 236.777027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.243842] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 241.179435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.542810] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 244.873983] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 246.710962] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 248.928392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.979323] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 253.267736] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 266.720039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 266.723784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.728462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 266.731205] Lustre: Skipped 4 previous similar messages [ 268.618692] Lustre: server umount lustre-OST0000 complete [ 272.031993] Lustre: server umount lustre-MDT0000 complete [ 272.353082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 272.355617] LustreError: Skipped 1 previous similar message [ 272.357568] LustreError: 19269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.263309] LustreError: 20003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574840 with bad export cookie 17478071620634505187 [ 273.266267] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.267237] LustreError: 20003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 273.379505] Lustre: server umount lustre-MDT0001 complete [ 277.185359] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 278.240780] Key type lgssc unregistered [ 278.379452] LNet: 22816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.383598] LNetError: 22816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.394378] LNet: Removed LNI 192.168.202.120@tcp [ 278.753109] Key type .llcrypt unregistered [ 278.754346] Key type ._llcrypt unregistered [ 287.456422] Key type ._llcrypt registered [ 287.457669] Key type .llcrypt registered [ 287.498633] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 287.805426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 287.838032] alg: No test for adler32 (adler32-zlib) [ 288.695535] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 288.785902] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 290.367232] Key type lgssc registered [ 290.715495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 293.271723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 293.280342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.425988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.762844] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 298.566481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.705939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.151914] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 302.664444] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 304.629570] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 306.859180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.989237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 308.898302] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 311.124843] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 320.301102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 329.697111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 329.700699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.704723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 330.722081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 330.725204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.726073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 330.731987] Lustre: Skipped 1 previous similar message [ 331.803800] Lustre: server umount lustre-OST0000 complete [ 334.815963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 334.818415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.822279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.231327] Lustre: server umount lustre-MDT0000 complete [ 340.480201] LustreError: 24182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574908 with bad export cookie 3028342645553583771 [ 340.483119] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.486041] LustreError: 24182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.595103] Lustre: server umount lustre-MDT0001 complete [ 344.289228] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 345.346587] Key type lgssc unregistered [ 345.468332] LNet: 28193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.470633] LNetError: 28193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.477296] LNet: Removed LNI 192.168.202.120@tcp [ 345.797774] Key type .llcrypt unregistered [ 345.798843] Key type ._llcrypt unregistered [ 354.708603] Key type ._llcrypt registered [ 354.709783] Key type .llcrypt registered [ 354.751926] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 355.142118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.181113] alg: No test for adler32 (adler32-zlib) [ 356.049968] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 356.149564] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 357.727144] Key type lgssc registered [ 358.093266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 360.642823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 360.648654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.790198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 363.097926] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 365.988767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.127594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 367.435756] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 369.904607] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 371.678611] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 375.856873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 375.970647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.667571] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 379.921646] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 387.239671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 392.676940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.678224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 392.678961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 392.680184] Lustre: Skipped 1 previous similar message [ 393.745737] Lustre: server umount lustre-OST0000 complete [ 396.768210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 396.770399] LustreError: Skipped 1 previous similar message [ 396.771619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.775562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.777360] Lustre: Skipped 1 previous similar message [ 397.087563] Lustre: server umount lustre-MDT0000 complete [ 397.791980] LustreError: 31824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.792554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.796693] LustreError: 31824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 398.271655] LustreError: 31762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574965 with bad export cookie 10694979840201207364 [ 398.276223] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.276751] LustreError: 31762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 398.391638] Lustre: server umount lustre-MDT0001 complete [ 402.224741] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 403.297608] Key type lgssc unregistered [ 403.426472] LNet: 33471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.429917] LNetError: 33471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 403.440361] LNet: Removed LNI 192.168.202.120@tcp [ 403.744127] Key type .llcrypt unregistered [ 403.745873] Key type ._llcrypt unregistered [ 412.581084] Key type ._llcrypt registered [ 412.582086] Key type .llcrypt registered [ 412.619693] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 413.168480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 413.193137] alg: No test for adler32 (adler32-zlib) [ 414.043861] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 414.141208] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 415.727138] Key type lgssc registered [ 416.100261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 418.753725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 418.760269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.912697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.242724] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 424.094566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.228328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 425.642118] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 428.092201] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 430.012549] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.410063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.537817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.549906] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 437.086001] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.866715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 451.041356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.042929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.045073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.048272] Lustre: Skipped 1 previous similar message [ 452.622107] Lustre: server umount lustre-OST0000 complete [ 455.136116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 455.139036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.143562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.145609] Lustre: Skipped 1 previous similar message [ 456.160834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.162106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.058403] Lustre: server umount lustre-MDT0000 complete [ 461.280122] LustreError: 34856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.285278] LustreError: 34856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 461.364246] LustreError: 34837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575029 with bad export cookie 8837643489545870134 [ 461.367887] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 461.368376] LustreError: 34837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 461.485187] Lustre: server umount lustre-MDT0001 complete [ 464.973845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 466.063662] Key type lgssc unregistered [ 466.187367] LNet: 38404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.189798] LNetError: 38404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.198315] LNet: Removed LNI 192.168.202.120@tcp [ 466.528426] Key type .llcrypt unregistered [ 466.529489] Key type ._llcrypt unregistered [ 474.860333] Key type ._llcrypt registered [ 474.861478] Key type .llcrypt registered [ 474.899748] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 475.270100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 475.276271] alg: No test for adler32 (adler32-zlib) [ 476.115165] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 476.199194] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 477.775200] Key type lgssc registered [ 478.121917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 480.601821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 480.607513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.756656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.960891] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 485.696298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.824133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.036389] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 489.308565] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.090675] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.270198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.404993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.360093] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 497.834738] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 506.726096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 510.943854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.946191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.951341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.969559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.970145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.970599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 513.375409] Lustre: lustre-OST0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 513.377802] Lustre: Skipped 1 previous similar message [ 516.501843] Lustre: server umount lustre-OST0000 complete [ 529.354547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.359465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.361402] Lustre: Skipped 1 previous similar message [ 534.496587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.499201] Lustre: Skipped 1 previous similar message [ 535.509833] Lustre: server umount lustre-MDT0000 complete [ 536.544104] LustreError: 39788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.548120] LustreError: 39788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 536.662054] LustreError: 39771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575104 with bad export cookie 13826259335558695470 [ 536.665067] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 536.666464] LustreError: 39771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 542.834295] Lustre: server umount lustre-MDT0001 complete [ 546.123129] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 547.030989] Key type lgssc unregistered [ 547.147303] LNet: 43508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.150867] LNetError: 43508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.164278] LNet: Removed LNI 192.168.202.120@tcp [ 547.438620] Key type .llcrypt unregistered [ 547.439691] Key type ._llcrypt unregistered [ 555.549487] Key type ._llcrypt registered [ 555.550365] Key type .llcrypt registered [ 555.583664] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 555.880218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 555.913564] alg: No test for adler32 (adler32-zlib) [ 556.755262] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 556.832512] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 558.407143] Key type lgssc registered [ 558.731957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.145377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 561.151334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.280255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.527174] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 566.132030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.253735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 567.514525] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 569.739948] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.415507] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.454944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 573.560213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.247142] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 577.447187] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.851569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 591.839940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.842514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.846745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.352428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.354637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.360093] Lustre: Skipped 1 previous similar message [ 596.570826] Lustre: server umount lustre-MDT0000 complete [ 597.472327] LustreError: 44891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.479430] LustreError: 44891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 597.811424] LustreError: 44872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575165 with bad export cookie 7534653929186754610 [ 597.815439] LustreError: 44872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 597.815467] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.360631] LustreError: 47139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.368418] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [ 598.370566] Lustre: Skipped 3 previous similar messages [ 602.591874] LustreError: 44886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.592154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.603819] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 607.712163] LustreError: 44915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.712394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 607.717849] LustreError: 44915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 607.719574] Lustre: Skipped 1 previous similar message [ 612.319110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.386342] Lustre: server umount lustre-MDT0001 complete [ 630.096379] Lustre: server umount lustre-OST0000 complete [ 635.498579] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 636.511696] Key type lgssc unregistered [ 636.627310] LNet: 48767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.629880] LNetError: 48767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.638272] LNet: Removed LNI 192.168.202.120@tcp [ 636.919879] Key type .llcrypt unregistered [ 636.920803] Key type ._llcrypt unregistered [ 646.723256] Key type ._llcrypt registered [ 646.724480] Key type .llcrypt registered [ 646.767099] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 647.156624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.200043] alg: No test for adler32 (adler32-zlib) [ 648.044151] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 648.122639] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 649.695132] Key type lgssc registered [ 650.052245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.629799] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.635846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.063454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 673.183331] LustreError: 50129:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 699.807179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.807761] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 704.242295] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 867.067028] Lustre: server umount lustre-OST0000 complete [ 872.910866] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 873.949207] Key type lgssc unregistered [ 874.089396] LNet: 51799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.092208] LNetError: 51799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.099369] LNet: Removed LNI 192.168.202.120@tcp [ 874.412727] Key type .llcrypt unregistered [ 874.413912] Key type ._llcrypt unregistered [ 882.697149] Key type ._llcrypt registered [ 882.698115] Key type .llcrypt registered [ 882.729712] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 882.987435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.992400] alg: No test for adler32 (adler32-zlib) [ 883.832395] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 883.912780] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 885.487138] Key type lgssc registered [ 885.813913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 888.165207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 888.171295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.333330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.651172] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 893.322505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.451027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.698737] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 896.888752] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 903.626337] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.306602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.421086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.316569] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 913.818185] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.599351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 918.817482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 918.818563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.841495] Lustre: server umount lustre-OST0000 complete [ 924.129068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.129341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 924.131901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 924.131905] Lustre: Skipped 1 previous similar message [ 924.134056] Lustre: Skipped 1 previous similar message [ 928.280165] Lustre: server umount lustre-MDT0000 complete [ 929.248226] LustreError: 53184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.254386] LustreError: 53184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 929.508663] LustreError: 53165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575497 with bad export cookie 15386465645075429729 [ 929.511563] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.512883] LustreError: 53165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 929.633620] Lustre: server umount lustre-MDT0001 complete [ 933.101307] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 934.077479] Key type lgssc unregistered [ 934.204377] LNet: 56710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.206430] LNetError: 56710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.212275] LNet: Removed LNI 192.168.202.120@tcp [ 934.493211] Key type .llcrypt unregistered [ 934.494143] Key type ._llcrypt unregistered [ 942.757769] Key type ._llcrypt registered [ 942.758644] Key type .llcrypt registered [ 942.797048] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 943.136878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.202387] alg: No test for adler32 (adler32-zlib) [ 944.039832] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 944.123322] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 945.695140] Key type lgssc registered [ 946.002326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.311477] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.316551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.743323] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 968.863246] LustreError: 58070:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 995.487192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.341904] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 999.630301] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1001.737456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.893763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.913606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1003.086794] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1005.738066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.858984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.140959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1009.427526] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1011.150842] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.192081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.195508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.199703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.267959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.269136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.930633] Lustre: server umount lustre-OST0000 complete [ 1032.050096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.051394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.053843] Lustre: Skipped 1 previous similar message [ 1032.055829] Lustre: Skipped 1 previous similar message [ 1036.192288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.194104] Lustre: Skipped 1 previous similar message [ 1038.233074] Lustre: server umount lustre-MDT0000 complete [ 1039.553102] LustreError: 58078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575607 with bad export cookie 12779968421483546363 [ 1039.556115] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.556785] LustreError: 58078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1045.746602] Lustre: server umount lustre-MDT0001 complete [ 1049.689606] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1050.875870] Key type lgssc unregistered [ 1051.018413] LNet: 61810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.021398] LNetError: 61810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.028903] LNet: Removed LNI 192.168.202.120@tcp [ 1051.385746] Key type .llcrypt unregistered [ 1051.386951] Key type ._llcrypt unregistered [ 1060.428505] Key type ._llcrypt registered [ 1060.429913] Key type .llcrypt registered [ 1060.471783] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1060.887631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.894951] alg: No test for adler32 (adler32-zlib) [ 1061.750750] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1061.842701] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1063.423155] Key type lgssc registered [ 1063.817057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.763826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1066.777782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.934131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.435425] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1072.653644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.783684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.181102] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1076.549252] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.294574] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1080.415825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.529630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.406332] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1084.856261] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.838566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1098.210860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.210932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.212204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.214064] LustreError: Skipped 1 previous similar message [ 1098.218639] Lustre: Skipped 1 previous similar message [ 1099.625665] Lustre: server umount lustre-OST0000 complete [ 1103.192345] Lustre: server umount lustre-MDT0000 complete [ 1103.328472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.329076] LustreError: 63189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.332916] Lustre: Skipped 1 previous similar message [ 1103.340953] LustreError: 63189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1104.545480] LustreError: 63175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575672 with bad export cookie 2744209758886052009 [ 1104.548378] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.550466] LustreError: 63175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1104.663559] Lustre: server umount lustre-MDT0001 complete [ 1108.366800] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1109.446315] Key type lgssc unregistered [ 1109.585367] LNet: 66741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.588728] LNetError: 66741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.596271] LNet: Removed LNI 192.168.202.120@tcp [ 1109.914915] Key type .llcrypt unregistered [ 1109.916038] Key type ._llcrypt unregistered [ 1119.955590] Key type ._llcrypt registered [ 1119.956764] Key type .llcrypt registered [ 1119.993880] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1120.337705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.359562] alg: No test for adler32 (adler32-zlib) [ 1121.217682] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1121.306542] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1122.887143] Key type lgssc registered [ 1123.312879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.780610] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:34:55 (1788575695) [ 1130.183776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1130.191077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.341486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.842322] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1135.849423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.979655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.545853] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1140.670386] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1144.080275] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1146.860458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1146.987349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.082646] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1151.783193] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1156.198659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1164.489095] Lustre: Failing over lustre-MDT0000 [ 1164.694653] Lustre: server umount lustre-MDT0000 complete [ 1166.816569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1166.817904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.820898] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.824901] Lustre: Skipped 1 previous similar message [ 1166.833373] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1168.313903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.363550] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.421923] Lustre: *** cfs_fail_loc=135, val=0*** [ 1168.466374] LustreError: 71522:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1168.468962] LustreError: 71522:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -2 [ 1168.472048] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1168.472048] [ 1168.476709] LustreError: MGC192.168.202.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1168.482845] LustreError: 71509:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1168.486318] LustreError: 71509:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 1168.489970] LustreError: 71509:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1168.492683] LustreError: 71509:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1168.503411] Lustre: server umount lustre-MDT0000 complete [ 1168.504866] LustreError: 71509:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1168.735860] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.021739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.077906] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.177057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.199774] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1172.580272] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1173.849241] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1176.547468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1176.558312] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1176.576932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1181.665988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.666734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.669386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.672365] Lustre: Skipped 2 previous similar messages [ 1181.676838] Lustre: Skipped 1 previous similar message [ 1183.756076] Lustre: server umount lustre-OST0000 complete [ 1186.785705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.785854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1186.786795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.789602] Lustre: Skipped 1 previous similar message [ 1191.190431] Lustre: server umount lustre-MDT0000 complete [ 1191.903960] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.908784] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1192.440171] LustreError: 68398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575760 with bad export cookie 5395374055417884447 [ 1192.442901] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.444968] LustreError: 68398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1192.555304] Lustre: server umount lustre-MDT0001 complete [ 1196.032249] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1196.979390] Key type lgssc unregistered [ 1197.111357] LNet: 73290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.114572] LNetError: 73290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.122274] LNet: Removed LNI 192.168.202.120@tcp [ 1197.438462] Key type .llcrypt unregistered [ 1197.439474] Key type ._llcrypt unregistered [ 1206.082052] Key type ._llcrypt registered [ 1206.083064] Key type .llcrypt registered [ 1206.120763] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1206.523123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.538117] alg: No test for adler32 (adler32-zlib) [ 1207.397928] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1207.491983] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1209.071234] Key type lgssc registered [ 1209.427538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.883548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.890551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.036483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.450739] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1217.236542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.346626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.593681] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1220.884165] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.647374] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1224.887015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1224.998382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.852421] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1229.184685] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1238.310602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1242.153046] Lustre: Failing over lustre-MDT0000 [ 1242.347550] Lustre: server umount lustre-MDT0000 complete [ 1243.103811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1243.106410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.110825] LustreError: 75401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.115639] LustreError: 75401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1243.617652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.618261] LustreError: 74670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.622397] Lustre: Skipped 1 previous similar message [ 1245.022825] LustreError: 74672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.028221] LustreError: 74672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1245.559411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.607452] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.666214] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1245.716369] LustreError: 77779:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1245.718196] LustreError: 77779:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -9 [ 1245.720515] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1245.720515] [ 1245.723758] LustreError: MGC192.168.202.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1245.728529] LustreError: 77767:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1245.731153] LustreError: 77767:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 1245.733863] LustreError: 77767:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1245.735857] LustreError: 77767:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1245.746384] Lustre: server umount lustre-MDT0000 complete [ 1245.747801] LustreError: 77767:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1248.012032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.051599] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.135698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.160019] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1249.492123] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1250.136704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1253.352073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1253.362268] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1253.378320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1257.951845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.954060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.957632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.466211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.468012] Lustre: Skipped 1 previous similar message [ 1260.363088] Lustre: server umount lustre-OST0000 complete [ 1263.584448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.584798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.586981] LustreError: Skipped 1 previous similar message [ 1263.592561] Lustre: Skipped 1 previous similar message [ 1263.596427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.598326] Lustre: Skipped 1 previous similar message [ 1267.799878] Lustre: server umount lustre-MDT0000 complete [ 1268.704168] LustreError: 74675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.709949] LustreError: 74675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1269.110396] LustreError: 76514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575836 with bad export cookie 3388144690665969313 [ 1269.113481] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.118279] LustreError: 76514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1269.244377] Lustre: server umount lustre-MDT0001 complete [ 1273.260601] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1274.438911] Key type lgssc unregistered [ 1274.587484] LNet: 79554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.591508] LNetError: 79554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.601293] LNet: Removed LNI 192.168.202.120@tcp [ 1274.960990] Key type .llcrypt unregistered [ 1274.962269] Key type ._llcrypt unregistered [ 1279.766514] Key type ._llcrypt registered [ 1279.767730] Key type .llcrypt registered [ 1279.809249] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1279.933419] Key type .llcrypt unregistered [ 1279.935296] Key type ._llcrypt unregistered [ 1289.364723] Key type ._llcrypt registered [ 1289.366213] Key type .llcrypt registered [ 1289.405478] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1289.833371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.844661] alg: No test for adler32 (adler32-zlib) [ 1290.695796] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1290.794077] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1292.383203] Key type lgssc registered [ 1292.849218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1295.886478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1295.897382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.078355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.595467] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1301.549529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.727158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.207164] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1306.187323] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1308.316975] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.103497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.246051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.517480] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1316.172355] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1321.511840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1326.564234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.564306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1326.569138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1326.572987] Lustre: Skipped 1 previous similar message [ 1330.844269] Lustre: server umount lustre-OST0000 complete [ 1332.703786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1332.706766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.711169] Lustre: Skipped 1 previous similar message [ 1332.713433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.802143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.805562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.815354] Lustre: Skipped 1 previous similar message [ 1338.755417] Lustre: server umount lustre-MDT0000 complete [ 1340.730056] LustreError: 83263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575908 with bad export cookie 10634785002927490335 [ 1340.734343] LustreError: 83263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1340.735663] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.854645] Lustre: server umount lustre-MDT0001 complete [ 1346.033374] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1347.418324] Key type lgssc unregistered [ 1347.577401] LNet: 84953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.579711] LNetError: 84953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.587742] LNet: Removed LNI 192.168.202.120@tcp [ 1347.925128] Key type .llcrypt unregistered [ 1347.927033] Key type ._llcrypt unregistered [ 1357.203826] Key type ._llcrypt registered [ 1357.205179] Key type .llcrypt registered [ 1357.244781] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1357.582479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.660349] alg: No test for adler32 (adler32-zlib) [ 1358.515621] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1358.607895] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1360.199142] Key type lgssc registered [ 1360.545651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.076327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.083976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.232372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.751616] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1368.942047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.095668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1375.574409] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1378.249870] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.250939] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1382.817682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1382.992492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.215581] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1388.313281] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1390.191266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 1399.776752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1399.780668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.785032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1399.792537] Lustre: Skipped 1 previous similar message [ 1399.797557] Lustre: Skipped 1 previous similar message [ 1403.357445] Lustre: server umount lustre-OST0000 complete [ 1410.016632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1410.016752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.017628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.027135] Lustre: Skipped 1 previous similar message [ 1415.136015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.138131] Lustre: Skipped 2 previous similar messages [ 1419.231128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1419.301196] Lustre: server umount lustre-MDT0000 complete [ 1420.257671] LustreError: 86351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.263671] LustreError: 86351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1420.574250] LustreError: 89130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575988 with bad export cookie 4847514584429671054 [ 1420.577633] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.577779] LustreError: 89130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1420.687863] Lustre: server umount lustre-MDT0001 complete [ 1425.565136] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1426.662251] Key type lgssc unregistered [ 1426.799341] LNet: 89881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.800949] LNetError: 89881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.807420] LNet: Removed LNI 192.168.202.120@tcp [ 1427.123242] Key type .llcrypt unregistered [ 1427.124280] Key type ._llcrypt unregistered [ 1437.342918] Key type ._llcrypt registered [ 1437.344017] Key type .llcrypt registered [ 1437.399380] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1437.736306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1437.771957] alg: No test for adler32 (adler32-zlib) [ 1438.624316] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1438.710701] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1440.295323] Key type lgssc registered [ 1440.845836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.521983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.529874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.707315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.371295] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1453.117956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.288470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1455.383561] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1458.800357] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.018560] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1464.084855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1464.240765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.624916] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1469.577370] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1471.400110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 1486.816180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.819248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.825599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.841046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.842040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.844626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.492199] Lustre: server umount lustre-OST0000 complete [ 1492.960846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.961787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.966317] Lustre: Skipped 1 previous similar message [ 1492.968440] Lustre: Skipped 1 previous similar message [ 1497.215391] Lustre: server umount lustre-MDT0000 complete [ 1498.081299] LustreError: 91260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.087016] LustreError: 91260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1499.128493] LustreError: 92001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576066 with bad export cookie 6787626128376215601 [ 1499.128866] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.135024] LustreError: 92001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1499.306617] Lustre: server umount lustre-MDT0001 complete [ 1504.160915] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1505.342831] Key type lgssc unregistered [ 1505.513218] LNet: 94817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.523163] LNetError: 94817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.533795] LNet: Removed LNI 192.168.202.120@tcp [ 1506.877391] Key type .llcrypt unregistered [ 1506.878524] Key type ._llcrypt unregistered [ 1522.712780] Key type ._llcrypt registered [ 1522.714356] Key type .llcrypt registered [ 1522.787794] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1523.484031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.570346] alg: No test for adler32 (adler32-zlib) [ 1524.581542] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1524.759163] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1526.416948] Key type lgssc registered [ 1527.254362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1531.962087] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1531.976447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1547.487982] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1552.607485] LustreError: 96178:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 1579.231174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.462268] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1589.347887] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1593.284918] Lustre: server umount lustre-OST0000 complete [ 1598.850789] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:42:45 (1788576165) [ 1603.771556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.100363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.577281] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1609.186973] LustreError: 97812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1609.200263] LustreError: 97812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1614.313672] LustreError: 97813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.964255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.183916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1618.252545] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1623.862660] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1629.269380] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1642.079685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.349434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.503848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 1648.849104] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1655.827014] hrtimer: interrupt took 4427752 ns [ 1656.119258] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1668.075946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.081458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.189199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.194838] Lustre: Skipped 2 previous similar messages [ 1674.471633] Lustre: server umount lustre-OST0000 complete [ 1681.889386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.898436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.909188] Lustre: Skipped 1 previous similar message [ 1681.917922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.427727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.546044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.564984] Lustre: Skipped 3 previous similar messages [ 1693.151074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1693.233250] Lustre: server umount lustre-MDT0000 complete [ 1693.667604] LustreError: 100465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.683553] LustreError: 100465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1697.765088] LustreError: 98542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.778790] LustreError: 98542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1702.884260] LustreError: 97808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.966981] LustreError: 97808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1704.522679] LustreError: 97796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576272 with bad export cookie 4142050323028387419 [ 1704.547460] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.553846] LustreError: 97796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1705.977535] Lustre: server umount lustre-MDT0001 complete [ 1729.276940] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1736.488173] Key type lgssc unregistered [ 1736.801937] LNet: 101530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.817102] LNetError: 101530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.850658] LNet: Removed LNI 192.168.202.120@tcp [ 1738.858884] Key type .llcrypt unregistered [ 1738.868293] Key type ._llcrypt unregistered [ 1793.044630] Key type ._llcrypt registered [ 1793.048989] Key type .llcrypt registered [ 1793.211732] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1794.308491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.466539] alg: No test for adler32 (adler32-zlib) [ 1795.537830] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1795.870375] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1797.619499] Key type lgssc registered [ 1798.713876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.036879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1806.074570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.669595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.930931] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1818.094270] LustreError: 103715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.121392] LustreError: 103715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1820.787300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.121925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1825.778915] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1833.918936] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1841.112949] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.121537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.439812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.529297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 1856.050217] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1865.097618] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.453169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.462230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.490990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.514496] Lustre: Skipped 1 previous similar message [ 1881.493873] Lustre: server umount lustre-OST0000 complete [ 1886.690663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.711131] Lustre: Skipped 1 previous similar message [ 1886.719909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.812366] LustreError: 103714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.837721] LustreError: 103714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1892.028625] Lustre: server umount lustre-MDT0000 complete [ 1896.167574] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576463 with bad export cookie 7500795632646052866 [ 1896.170054] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.186525] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1896.688670] Lustre: server umount lustre-MDT0001 complete [ 1909.818602] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1913.483879] Key type lgssc unregistered [ 1913.964499] LNet: 107248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.970147] LNetError: 107248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.988761] LNet: Removed LNI 192.168.202.120@tcp [ 1914.817951] Key type .llcrypt unregistered [ 1914.825950] Key type ._llcrypt unregistered [ 1935.007205] Key type ._llcrypt registered [ 1935.011039] Key type .llcrypt registered [ 1935.179123] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1936.041256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.075397] alg: No test for adler32 (adler32-zlib) [ 1937.649893] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 1937.913534] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1939.679263] Key type lgssc registered [ 1941.127842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.565495] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.581996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1965.151606] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1970.271753] LustreError: 108484:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 1996.969547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.252798] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2014.116630] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2021.275220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.577376] LustreError: MGC192.168.202.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2021.584805] LustreError: 109510:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2021.589394] LustreError: 109510:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2021.594265] LustreError: 109510:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2021.693451] Lustre: server umount lustre-MDT0000 complete [ 2021.701224] LustreError: 109510:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2040.092327] Lustre: server umount lustre-OST0000 complete [ 2053.040189] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2064.447603] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2107.387718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.738179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2107.774924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2107.866776] Lustre: lustre-MDT0000: new disk, initializing [ 2107.995128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.018758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2113.927791] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2127.087788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.433944] Lustre: 112304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2127.569816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2127.576303] Lustre: Skipped 1 previous similar message [ 2127.747757] Lustre: lustre-MDT0001: new disk, initializing [ 2127.938923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2127.974064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2127.996656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2133.456476] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2145.148490] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.173814] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.532360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2159.982523] Lustre: lustre-OST0000: new disk, initializing [ 2159.993858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2160.005906] Lustre: 114274:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2160.171458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.962729] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2168.417888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2168.427868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2168.602150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2179.516993] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2185.351167] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2185.594509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2191.093733] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2191.270977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2194.402938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.408329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2194.437570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2194.459461] Lustre: Skipped 1 previous similar message [ 2199.524399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2199.533934] Lustre: Skipped 1 previous similar message [ 2199.851116] Lustre: server umount lustre-OST0000 complete [ 2204.640848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.654362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.670234] Lustre: Skipped 2 previous similar messages [ 2209.762708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2209.765658] Lustre: Skipped 2 previous similar messages [ 2211.092224] Lustre: server umount lustre-MDT0000 complete [ 2215.506667] LustreError: 113099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576783 with bad export cookie 185937981207287491 [ 2215.510180] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.528300] LustreError: 113099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2215.836957] Lustre: server umount lustre-MDT0001 complete [ 2224.986931] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:53:11 (1788576791) [ 2229.194687] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2248.721672] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2259.836807] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2310.348350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.778241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2310.849643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2311.008542] Lustre: lustre-MDT0000: new disk, initializing [ 2311.155975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.176225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2317.038308] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2328.847132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.955755] Lustre: 119177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2329.007488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2329.013444] Lustre: Skipped 1 previous similar message [ 2329.082491] Lustre: lustre-MDT0001: new disk, initializing [ 2329.229165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.291098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2329.311548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2335.482673] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2347.044930] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2354.343940] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2362.668575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.036963] Lustre: lustre-OST0000: new disk, initializing [ 2363.044783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2363.054857] Lustre: 121146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2363.218075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.736945] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2370.693118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2370.708751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2370.794931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2380.698464] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2388.003672] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2388.595643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2396.246400] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2396.715191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2401.762353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2401.766888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.779116] LustreError: Skipped 1 previous similar message [ 2401.780132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.831659] Lustre: Skipped 1 previous similar message [ 2405.660453] Lustre: server umount lustre-OST0000 complete [ 2410.977093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2410.982272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.990768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.994845] Lustre: Skipped 1 previous similar message [ 2416.771429] Lustre: server umount lustre-MDT0000 complete [ 2417.122924] LustreError: 120511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.149519] LustreError: 120511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2421.250268] LustreError: 119168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576988 with bad export cookie 185937981207288373 [ 2421.261491] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2421.263577] LustreError: 119168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2421.719418] Lustre: server umount lustre-MDT0001 complete [ 2431.634108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.104919] LustreError: 123363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2432.188838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.600836] LustreError: 123364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.168405] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2441.702242] LustreError: 123363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.822578] LustreError: 123364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.341831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.295753] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2463.457536] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2471.764727] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2480.689523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2488.543426] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2497.988240] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2509.810693] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2516.964594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.975250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2517.000928] Lustre: Skipped 2 previous similar messages [ 2517.024564] Lustre: Skipped 3 previous similar messages [ 2521.332822] Lustre: server umount lustre-OST0000 complete [ 2527.207779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.218791] Lustre: Skipped 1 previous similar message [ 2532.137220] Lustre: server umount lustre-MDT0000 complete [ 2532.322082] LustreError: 124791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.344825] LustreError: 124791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2536.755040] LustreError: 123346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577104 with bad export cookie 185937981207289255 [ 2536.765371] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2536.767307] LustreError: 123346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2537.627422] Lustre: server umount lustre-MDT0001 complete [ 2550.031699] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2552.515493] Key type lgssc unregistered [ 2552.836770] LNet: 126995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.843525] LNetError: 126995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.854648] LNet: Removed LNI 192.168.202.120@tcp [ 2553.911822] Key type .llcrypt unregistered [ 2553.913956] Key type ._llcrypt unregistered [ 2575.907471] Key type ._llcrypt registered [ 2575.909916] Key type .llcrypt registered [ 2576.097833] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2591.637689] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2592.943399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.182451] alg: No test for adler32 (adler32-zlib) [ 2594.250076] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 2594.596691] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2596.295193] Key type lgssc registered [ 2597.957100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.706976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.755153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.100207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2644.155735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2644.280134] Lustre: lustre-MDT0000: new disk, initializing [ 2644.381206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.407749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2649.839676] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2662.198574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.295887] Lustre: 130144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2662.359874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2662.364620] Lustre: Skipped 1 previous similar message [ 2662.500134] Lustre: lustre-MDT0001: new disk, initializing [ 2662.700285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2662.775426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2662.780508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2668.739898] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2679.711562] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.682417] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2695.026052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.278468] Lustre: lustre-OST0000: new disk, initializing [ 2695.281594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2695.288458] Lustre: 132129:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2695.374629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.154818] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2705.000627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2705.022253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2705.193859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2714.484757] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.960643] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2719.281372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2724.169209] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2724.535607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2731.005100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2731.009742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.012605] LustreError: Skipped 1 previous similar message [ 2731.035824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2731.042782] Lustre: Skipped 1 previous similar message [ 2736.109243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2736.119711] Lustre: Skipped 1 previous similar message [ 2741.216268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2741.237762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.249757] Lustre: Skipped 1 previous similar message [ 2741.452581] Lustre: server umount lustre-OST0000 complete [ 2746.336588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.351772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.364390] Lustre: Skipped 2 previous similar messages [ 2746.383066] Lustre: Skipped 1 previous similar message [ 2751.457547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.463908] Lustre: Skipped 1 previous similar message [ 2760.159147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2760.378410] Lustre: server umount lustre-MDT0000 complete [ 2764.973063] LustreError: 130137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577332 with bad export cookie 3244934793556048830 [ 2764.980196] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.993979] LustreError: 130137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2765.440416] Lustre: server umount lustre-MDT0001 complete [ 2775.160458] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:02:21 (1788577341) [ 2782.369665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.808457] LustreError: 134544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.831075] LustreError: 134544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2782.876919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.324440] LustreError: 134545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.366745] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2792.429038] LustreError: 134544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.546331] LustreError: 134545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.512832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.081147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2803.907872] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2812.728465] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2819.028822] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2824.231832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.237735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.245348] Lustre: Skipped 1 previous similar message [ 2824.257713] Lustre: Skipped 3 previous similar messages [ 2827.656613] Lustre: server umount lustre-MDT0000 complete [ 2829.282343] LustreError: 135272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.294557] LustreError: 135272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2831.894411] LustreError: 134524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577399 with bad export cookie 3244934793556049712 [ 2831.901055] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.911029] LustreError: 134524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2832.376292] Lustre: server umount lustre-MDT0001 complete [ 2840.849664] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:03:27 (1788577407) [ 2847.909858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.519805] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2868.639784] LustreError: 136959:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 2894.367776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2901.723347] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2909.049238] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2911.622034] Lustre: server umount lustre-OST0000 complete [ 2921.131565] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:04:47 (1788577487) [ 2928.556164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.107819] LustreError: 138407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.237821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.387942] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2944.082076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.202471] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2959.440773] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2966.010460] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2973.149626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.445652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.450127] Lustre: Skipped 1 previous similar message [ 2980.930693] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2989.135365] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.267474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.277276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2999.289653] Lustre: Skipped 1 previous similar message [ 2999.307706] Lustre: Skipped 2 previous similar messages [ 3003.528841] Lustre: server umount lustre-MDT0000 complete [ 3004.389530] LustreError: 139135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.408551] LustreError: 139135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3007.735166] LustreError: 138388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577575 with bad export cookie 3244934793556050377 [ 3007.751426] LustreError: 138388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3007.751829] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.215658] Lustre: server umount lustre-MDT0001 complete [ 3023.012091] Lustre: server umount lustre-OST0000 complete [ 3032.528747] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:06:38 (1788577598) [ 3039.100605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.431950] LustreError: 142062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.495359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.222896] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3053.942309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.361732] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3067.823497] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3073.350507] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3080.307716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3084.750090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3087.063566] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3096.205988] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3100.819845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3101.046815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3105.955986] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3106.200475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3110.380361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.385402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.393975] LustreError: Skipped 1 previous similar message [ 3110.395101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.443700] Lustre: Skipped 2 previous similar messages [ 3114.478967] Lustre: server umount lustre-OST0000 complete [ 3120.608144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.651604] Lustre: Skipped 1 previous similar message [ 3125.208516] Lustre: server umount lustre-MDT0000 complete [ 3125.735308] LustreError: 142063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.765643] LustreError: 142063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3129.817756] LustreError: 142044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577697 with bad export cookie 3244934793556051441 [ 3129.827612] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.832577] LustreError: 142044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3130.502391] Lustre: server umount lustre-MDT0001 complete [ 3139.395264] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:08:26 (1788577706) [ 3146.095359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.825666] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3166.943567] LustreError: 146222:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 3192.671473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.676839] Lustre: Skipped 2 previous similar messages [ 3199.749691] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3208.256210] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3215.792535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.157956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3220.392798] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3229.668419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.603920] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3244.836552] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3251.220413] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3255.533434] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3255.721661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3260.001275] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3260.179367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3266.024335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.025151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.026460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.026466] Lustre: Skipped 3 previous similar messages [ 3266.116376] Lustre: Skipped 2 previous similar messages [ 3268.840772] Lustre: server umount lustre-MDT0000 complete [ 3271.140923] LustreError: 147259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.168884] LustreError: 147259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3273.986201] LustreError: 146229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577841 with bad export cookie 3244934793556052470 [ 3274.002250] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3274.005718] LustreError: 146229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3274.703265] Lustre: server umount lustre-MDT0001 complete [ 3291.524347] Lustre: server umount lustre-OST0000 complete [ 3300.430490] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:11:06 (1788577866) [ 3307.723169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3323.231710] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3328.352509] LustreError: 150417:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 3360.443261] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3369.766500] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3377.490852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.054833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3382.727409] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3391.948321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.122364] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3405.056404] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3410.946057] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3418.477342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.733894] Lustre: lustre-OST0001: new disk, initializing [ 3418.738493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3418.747657] Lustre: 153293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3424.313535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3424.322972] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3424.489105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3425.720509] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3436.026440] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.651444] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3440.836802] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3446.518282] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3446.881748] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3449.829754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.845161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.889514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3449.895916] Lustre: Skipped 2 previous similar messages [ 3455.426677] Lustre: server umount lustre-OST0000 complete [ 3460.577609] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3474.399749] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3474.610512] Lustre: server umount lustre-OST0001 complete [ 3479.529696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.542327] LustreError: Skipped 1 previous similar message [ 3493.343114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3493.720950] Lustre: server umount lustre-MDT0000 complete [ 3497.631715] LustreError: 150425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578065 with bad export cookie 3244934793556053422 [ 3497.635072] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.641654] LustreError: 150425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3497.937849] Lustre: server umount lustre-MDT0001 complete [ 3522.034915] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:14:48 (1788578088) [ 3523.542712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3525.438691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3527.321752] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:14:53 (1788578093) [ 3534.596296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.984050] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3535.024688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3535.272340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.275424] Lustre: Skipped 6 previous similar messages [ 3540.459466] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3549.156167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.265772] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3549.286315] Lustre: 156855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3555.631589] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3568.373411] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3576.444766] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3584.431162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.676412] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3592.126466] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3594.477369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3600.901415] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3606.690481] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3606.936659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3612.329657] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3612.595502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3615.200356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.203783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.231018] LustreError: Skipped 1 previous similar message [ 3615.249965] Lustre: Skipped 6 previous similar messages [ 3621.155473] Lustre: server umount lustre-OST0000 complete [ 3644.104747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.747775] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3645.775632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3645.779078] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3645.813928] Lustre: Skipped 1 previous similar message [ 3651.550537] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3660.125855] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3665.975525] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3666.257312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3671.776941] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3672.094619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3679.052602] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff904686f51000.ost_server_uuid 50 [ 3681.200452] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff904686f51000.ost_server_uuid in FULL state after 0 sec [ 3681.407030] LustreError: 158581:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.20@tcp) returned error from blocking AST (req@00000000ea99ecee x1875459086494592 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b9d5be15/0x2d0853009936d708 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.20@tcp remote: 0xa8d1c23921197d88 expref: 5 pid: 158581 timeout: 3781 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3681.437494] LustreError: lustre-MDT0000: A client on nid 192.168.202.20@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3681.443064] LustreError: 156851:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.20@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b9d5be15/0x2d0853009936d708 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.20@tcp remote: 0xa8d1c23921197d88 expref: 6 pid: 158581 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3686.893034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.640452] Lustre: server umount lustre-OST0000 complete [ 3707.364807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3707.377183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.414457] Lustre: Skipped 27 previous similar messages [ 3707.650827] Lustre: server umount lustre-MDT0000 complete [ 3712.485484] LustreError: 158581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.521779] LustreError: 158581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 3712.566975] LustreError: 156846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578280 with bad export cookie 3244934793556054605 [ 3712.569666] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.597487] LustreError: 156846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3719.223180] Lustre: server umount lustre-MDT0001 complete [ 3732.727711] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3735.964182] Key type lgssc unregistered [ 3736.255137] LNet: 162981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3736.269152] LNetError: 162981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3736.313219] LNet: Removed LNI 192.168.202.120@tcp [ 3737.523462] Key type .llcrypt unregistered [ 3737.526450] Key type ._llcrypt unregistered [ 3764.433636] Key type ._llcrypt registered [ 3764.436254] Key type .llcrypt registered [ 3764.594772] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3765.769920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3765.799820] alg: No test for adler32 (adler32-zlib) [ 3766.899171] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 3767.125938] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3768.767572] Key type lgssc registered [ 3769.644783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3779.788239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3779.864095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.376931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.331976] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3790.819980] LustreError: 164391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3790.840576] LustreError: 164391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3795.962981] LustreError: 164390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3799.122049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.687592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3806.379405] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3816.409045] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3823.788310] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3831.079312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3831.669730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.918385] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3847.920835] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3851.192107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3859.233571] Lustre: Failing over lustre-MDT0000 [ 3859.525898] Lustre: server umount lustre-MDT0000 complete [ 3859.830687] LustreError: 164387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.970019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.978842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.482104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.508182] Lustre: Skipped 1 previous similar message [ 3866.594320] LustreError: 165318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.624885] LustreError: 165318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3871.722074] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.756231] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3871.986838] LustreError: 164371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.20@tcp arrived at 1788578439 with bad export cookie 1916835926929587648 [ 3877.856424] Lustre: 163700:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578429/real 1788578429] req@00000000478abaf8 x1875460315183232/t0(0) o400->MGC192.168.202.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788578445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3877.900565] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.777214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.976604] LustreError: 165318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.995350] LustreError: 165318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3888.586938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.688547] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3893.739927] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3893.765349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3893.965897] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3896.930366] Lustre: lustre-MDT0000: Denying connection for new client a735ca88-444b-4020-9980-0fcdd65ef95f (at 192.168.202.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 3906.535730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3906.547726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.562505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.096348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.098364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.393386] Lustre: server umount lustre-OST0000 complete [ 3914.527395] LustreError: 168121:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3914.536778] Lustre: 167471:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3914.553073] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3914.571730] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3914.590888] LustreError: 167471:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008066defa x1875460315203968/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3914.613882] LustreError: 167471:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3914.622831] LustreError: 167471:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3914.633966] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 3914.888785] Lustre: server umount lustre-MDT0000 complete [ 3919.104621] LustreError: 164370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578486 with bad export cookie 1916835926929587977 [ 3919.120203] LustreError: 164370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3919.126511] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.136494] LustreError: 163699:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3921.567168] Lustre: server umount lustre-MDT0001 complete [ 3932.549831] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3936.065185] Key type lgssc unregistered [ 3936.503096] LNet: 168874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3936.526786] LNetError: 168874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3936.553315] LNet: Removed LNI 192.168.202.120@tcp [ 3937.688451] Key type .llcrypt unregistered [ 3937.690226] Key type ._llcrypt unregistered [ 3964.200595] Key type ._llcrypt registered [ 3964.203302] Key type .llcrypt registered [ 3964.314231] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3965.436406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.464042] alg: No test for adler32 (adler32-zlib) [ 3966.676307] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 3966.907312] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3968.575183] Key type lgssc registered [ 3969.736441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3979.931344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3979.953381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.633858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.054327] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3992.051778] LustreError: 170296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3992.079485] LustreError: 170296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3996.858447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.153307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3997.321475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.453702] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4012.446670] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4019.542469] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4028.740151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4029.486211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.760602] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4047.027809] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.987556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4062.177417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4062.192082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.228071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4063.207320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4063.226729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.263643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4063.272557] Lustre: Skipped 1 previous similar message [ 4066.775308] Lustre: server umount lustre-OST0000 complete [ 4073.442127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.455634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.436193] Lustre: server umount lustre-MDT0000 complete [ 4078.560600] LustreError: 170296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.575532] LustreError: 170296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4081.788313] LustreError: 170276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578649 with bad export cookie 8077318820150974421 [ 4081.808186] LustreError: 170276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4081.814903] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4082.395658] Lustre: server umount lustre-MDT0001 complete [ 4094.918959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4098.342597] Key type lgssc unregistered [ 4098.685736] LNet: 173844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4098.697228] LNetError: 173844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4098.713831] LNet: Removed LNI 192.168.202.120@tcp [ 4099.794329] Key type .llcrypt unregistered [ 4099.797925] Key type ._llcrypt unregistered [ 4133.908942] Key type ._llcrypt registered [ 4133.911780] Key type .llcrypt registered [ 4134.018630] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4135.820700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4135.846339] alg: No test for adler32 (adler32-zlib) [ 4137.108491] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 4137.547547] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4139.279485] Key type lgssc registered [ 4140.890930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4150.254847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4150.265889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.736593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.006269] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4162.021538] LustreError: 175549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.048467] LustreError: 175549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4167.137944] LustreError: 175548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.811881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.351697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4174.118150] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4183.983931] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.939916] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4196.928361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.349460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.499878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4203.451698] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4210.804467] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4222.950119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.954695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4222.968460] Lustre: Skipped 1 previous similar message [ 4222.971942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4225.688042] Lustre: server umount lustre-OST0000 complete [ 4233.187519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.198729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4233.223371] Lustre: Skipped 1 previous similar message [ 4233.235560] Lustre: Skipped 2 previous similar messages [ 4238.279759] Lustre: server umount lustre-MDT0000 complete [ 4238.310215] LustreError: 175548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.340224] LustreError: 175548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4242.401364] LustreError: 175545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.426901] LustreError: 175545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4243.382602] LustreError: 177392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578811 with bad export cookie 1656317857667020387 [ 4243.384996] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.389144] LustreError: 177392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4243.617465] Lustre: server umount lustre-MDT0001 complete [ 4256.269885] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4259.382774] Key type lgssc unregistered [ 4259.798825] LNet: 179081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4259.814105] LNetError: 179081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4259.839993] LNet: Removed LNI 192.168.202.120@tcp [ 4261.214895] Key type .llcrypt unregistered [ 4261.221073] Key type ._llcrypt unregistered [ 4286.370401] Key type ._llcrypt registered [ 4286.376748] Key type .llcrypt registered [ 4286.521060] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4288.146598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4288.349879] alg: No test for adler32 (adler32-zlib) [ 4289.560295] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 4289.855495] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4291.551204] Key type lgssc registered [ 4292.929295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4305.295365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4305.321382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.822803] Lustre: *** cfs_fail_loc=135, val=0*** [ 4306.900520] LustreError: 180604:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4306.909488] LustreError: 180604:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -2 [ 4306.921518] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4306.921518] [ 4306.929752] LustreError: MGC192.168.202.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4306.940393] LustreError: 180566:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4306.947196] LustreError: 180566:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 4306.952184] LustreError: 180566:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 4306.961602] LustreError: 180566:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4307.029155] Lustre: server umount lustre-MDT0000 complete [ 4307.033732] LustreError: 180566:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4314.749778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.212783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.056502] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4325.349395] LustreError: 180975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.376298] LustreError: 180975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4330.376893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.475070] LustreError: 180976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.069024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.806559] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4347.599211] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4355.374345] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4360.685212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.705439] Lustre: Skipped 1 previous similar message [ 4360.705482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4364.401567] Lustre: server umount lustre-MDT0000 complete [ 4365.792976] LustreError: 180976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.810931] LustreError: 180976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4368.661419] LustreError: 181709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578936 with bad export cookie 7498602612647873408 [ 4368.675597] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.681940] LustreError: 181709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4369.076859] Lustre: server umount lustre-MDT0001 complete [ 4384.350661] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4387.542988] Key type lgssc unregistered [ 4387.909578] LNet: 183299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.929288] LNetError: 183299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4388.966079] LNet: Removed LNI 192.168.202.120@tcp [ 4389.898323] Key type .llcrypt unregistered [ 4389.901257] Key type ._llcrypt unregistered [ 4418.628873] Key type ._llcrypt registered [ 4418.632876] Key type .llcrypt registered [ 4418.728707] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4419.125406] Key type .llcrypt unregistered [ 4419.127509] Key type ._llcrypt unregistered [ 4441.061483] Key type ._llcrypt registered [ 4441.062936] Key type .llcrypt registered [ 4441.153370] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4442.367616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4442.475186] alg: No test for adler32 (adler32-zlib) [ 4443.863640] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 4444.228110] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4445.959314] Key type lgssc registered [ 4447.282661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4455.623118] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4455.659252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.199471] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4476.319470] LustreError: 185423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 4502.943757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.948522] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4520.727362] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4530.068259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.270633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.369146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4536.301691] LustreError: 185449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.331625] LustreError: 185449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4538.172560] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4538.354133] LustreError: 185448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.415587] LustreError: 185448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4543.467080] LustreError: 185449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.492698] LustreError: 185449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4548.583094] LustreError: 186461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.607473] LustreError: 186461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4551.100088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.602867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4556.638502] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4565.363197] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4573.184138] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4575.886531] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4587.500669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4587.506812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.516990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4587.533088] Lustre: Skipped 1 previous similar message [ 4587.543365] Lustre: Skipped 1 previous similar message [ 4590.399729] Lustre: server umount lustre-OST0000 complete [ 4597.729708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.743201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4597.746622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4597.746630] LustreError: Skipped 1 previous similar message [ 4597.769391] Lustre: Skipped 1 previous similar message [ 4597.809134] Lustre: Skipped 1 previous similar message [ 4601.187679] Lustre: server umount lustre-MDT0000 complete [ 4602.859842] LustreError: 186916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.879896] LustreError: 186916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4605.249148] LustreError: 185431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579172 with bad export cookie 15660653056045944163 [ 4605.254288] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.256290] LustreError: 185431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4605.567372] Lustre: server umount lustre-MDT0001 complete [ 4619.231547] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4622.461105] Key type lgssc unregistered [ 4622.875790] LNet: 189342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4622.891593] LNetError: 189342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4622.914136] LNet: Removed LNI 192.168.202.120@tcp [ 4623.912208] Key type .llcrypt unregistered [ 4623.913379] Key type ._llcrypt unregistered [ 4650.712928] Key type ._llcrypt registered [ 4650.715466] Key type .llcrypt registered [ 4650.857499] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4652.219989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4652.294505] alg: No test for adler32 (adler32-zlib) [ 4653.390921] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 4653.592443] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4655.287176] Key type lgssc registered [ 4656.746417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.571099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4665.638515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.643539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.404778] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4678.118859] LustreError: 190751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4678.138115] LustreError: 190751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4682.960669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.243161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4683.560339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4687.670775] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4698.060683] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4705.567912] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4714.543651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4714.910919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.691544] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4732.052682] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4737.470372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4742.576059] Lustre: Failing over lustre-MDT0000 [ 4742.631241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.636424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.648986] Lustre: Skipped 2 previous similar messages [ 4742.677701] Lustre: Skipped 2 previous similar messages [ 4743.074264] Lustre: server umount lustre-MDT0000 complete [ 4744.067773] LustreError: 193051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.748667] LustreError: 190751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.774329] LustreError: 190751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4752.868561] LustreError: 190746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4752.891039] LustreError: 190746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4757.985744] LustreError: 190745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.004588] LustreError: 190745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4763.353552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.576571] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4763.900610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.943932] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4764.512565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4768.979822] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4769.314185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4769.348922] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4769.386709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4774.462527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4782.735854] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4799.972121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.980473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.004532] Lustre: Skipped 1 previous similar message [ 4800.013401] Lustre: Skipped 1 previous similar message [ 4804.698545] Lustre: server umount lustre-OST0000 complete [ 4810.211055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.239332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4810.255803] Lustre: Skipped 1 previous similar message [ 4810.256206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4810.298590] Lustre: Skipped 1 previous similar message [ 4815.331341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.344885] Lustre: Skipped 1 previous similar message [ 4816.547453] Lustre: server umount lustre-MDT0000 complete [ 4819.436249] LustreError: 191783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.465247] LustreError: 191783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4821.677039] LustreError: 194581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579389 with bad export cookie 13365222771715066858 [ 4821.687507] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.698046] LustreError: 194581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4822.161584] Lustre: server umount lustre-MDT0001 complete [ 4836.621669] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4840.058284] Key type lgssc unregistered [ 4840.472734] LNet: 195660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4840.480478] LNetError: 195660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4840.498722] LNet: Removed LNI 192.168.202.120@tcp [ 4841.494597] Key type .llcrypt unregistered [ 4841.496481] Key type ._llcrypt unregistered [ 4871.580461] Key type ._llcrypt registered [ 4871.582623] Key type .llcrypt registered [ 4871.730929] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4873.350565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4873.415268] alg: No test for adler32 (adler32-zlib) [ 4874.546497] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 4874.880776] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4876.575354] Key type lgssc registered [ 4877.860445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4887.912629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4887.944972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.515675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.184443] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4899.821740] LustreError: 197071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.853271] LustreError: 197071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4904.931569] LustreError: 197072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.664463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.155647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4911.592265] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4920.774674] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4927.588662] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4936.209295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4936.751635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.597942] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4951.872880] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4960.378900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4967.064959] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4969.527240] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4989.427603] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5006.816579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5006.824599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.840367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5006.840711] Lustre: Skipped 1 previous similar message [ 5006.868699] Lustre: Skipped 1 previous similar message [ 5009.037698] Lustre: server umount lustre-OST0000 complete [ 5013.472436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5013.480700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.493740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5017.057554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5017.070865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.774565] Lustre: server umount lustre-MDT0000 complete [ 5022.178573] LustreError: 199692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.228169] LustreError: 199692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5024.260827] LustreError: 199908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579591 with bad export cookie 10092880913155408526 [ 5024.263491] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5024.274475] LustreError: 199908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5024.619833] Lustre: server umount lustre-MDT0001 complete [ 5037.031332] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5040.510679] Key type lgssc unregistered [ 5040.898975] LNet: 200984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5040.905963] LNetError: 200984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5040.928036] LNet: Removed LNI 192.168.202.120@tcp [ 5042.133870] Key type .llcrypt unregistered [ 5042.135481] Key type ._llcrypt unregistered [ 5062.686663] Key type ._llcrypt registered [ 5062.688426] Key type .llcrypt registered [ 5062.836637] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5064.213284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5064.329922] alg: No test for adler32 (adler32-zlib) [ 5065.477329] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 5065.779573] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5067.506086] Key type lgssc registered [ 5069.283218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5079.260875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5079.304309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.940045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.470476] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5091.303755] LustreError: 202248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.335683] LustreError: 202248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5096.419470] LustreError: 202249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.787495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.282832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5102.545689] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5112.397863] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5120.920476] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5129.422614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.149604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.082928] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5149.855285] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5150.684377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5157.275313] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5168.602836] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5179.714882] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5186.537632] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5197.796993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.815094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.852637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5201.889804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5201.893461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5201.900326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.931856] Lustre: Skipped 1 previous similar message [ 5203.845606] Lustre: server umount lustre-OST0000 complete [ 5210.080668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5210.094061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.122549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5212.131064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5212.146710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.997119] Lustre: server umount lustre-MDT0000 complete [ 5217.248618] LustreError: 202977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.269655] LustreError: 202977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5219.096391] LustreError: 202228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579786 with bad export cookie 14606209253289262169 [ 5219.121328] LustreError: 202228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5219.122362] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.542560] Lustre: server umount lustre-MDT0001 complete [ 5234.566739] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5237.657581] Key type lgssc unregistered [ 5238.018769] LNet: 206985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5238.033124] LNetError: 206985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5238.051603] LNet: Removed LNI 192.168.202.120@tcp [ 5239.482630] Key type .llcrypt unregistered [ 5239.484430] Key type ._llcrypt unregistered [ 5271.700151] Key type ._llcrypt registered [ 5271.702247] Key type .llcrypt registered [ 5271.894578] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5274.113580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5274.131578] alg: No test for adler32 (adler32-zlib) [ 5275.512223] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 5275.963897] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5277.783272] Key type lgssc registered [ 5279.236993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5287.956214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5287.973402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.336701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.401284] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5298.662893] LustreError: 208395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.708200] LustreError: 208395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5303.806432] LustreError: 208396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.150663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.886178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5310.863829] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5322.315581] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5330.452567] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5337.153961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5337.586164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5343.413295] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5354.431738] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5359.105900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5379.573302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.582108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5379.588334] Lustre: Skipped 1 previous similar message [ 5379.598503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.608328] Lustre: Skipped 1 previous similar message [ 5383.648375] Lustre: server umount lustre-OST0000 complete [ 5389.807234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.812350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5389.842389] Lustre: Skipped 1 previous similar message [ 5394.912461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5394.933108] Lustre: Skipped 2 previous similar messages [ 5400.049234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5400.056670] Lustre: Skipped 1 previous similar message [ 5404.127414] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5404.566038] Lustre: server umount lustre-MDT0000 complete [ 5405.163412] LustreError: 209124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.181645] LustreError: 209124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5408.974439] LustreError: 209131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579976 with bad export cookie 9759725666823336381 [ 5408.991981] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.999850] LustreError: 209131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5409.453983] Lustre: server umount lustre-MDT0001 complete [ 5422.624119] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5426.469197] Key type lgssc unregistered [ 5426.757473] LNet: 212105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5426.763725] LNetError: 212105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5426.777737] LNet: Removed LNI 192.168.202.120@tcp [ 5427.681734] Key type .llcrypt unregistered [ 5427.684395] Key type ._llcrypt unregistered [ 5448.570414] Key type ._llcrypt registered [ 5448.572897] Key type .llcrypt registered [ 5448.705985] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5449.827549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5450.010537] alg: No test for adler32 (adler32-zlib) [ 5451.184768] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 5451.477075] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5453.199197] Key type lgssc registered [ 5454.785389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5465.770425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5465.788267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.446904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5473.236421] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5477.881518] LustreError: 213369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5477.895433] LustreError: 213369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5482.979649] LustreError: 213370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.073674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.719696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.524860] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5500.573810] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5508.919616] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5517.913582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5518.413377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.551919] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5535.904195] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5537.925875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5558.759755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.765949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.772267] Lustre: Skipped 1 previous similar message [ 5558.794309] Lustre: Skipped 1 previous similar message [ 5563.891195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.909455] Lustre: Skipped 1 previous similar message [ 5564.673975] Lustre: server umount lustre-OST0000 complete [ 5571.556071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5571.570211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.613939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.113038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.126418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.168207] Lustre: server umount lustre-MDT0000 complete [ 5579.239155] LustreError: 213365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.247692] LustreError: 213365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5581.124055] LustreError: 213352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580148 with bad export cookie 17401443592028401008 [ 5581.142277] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5581.402784] Lustre: server umount lustre-MDT0001 complete [ 5594.332383] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5599.297497] Key type lgssc unregistered [ 5599.679762] LNet: 217084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.690949] LNetError: 217084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5599.719302] LNet: Removed LNI 192.168.202.120@tcp [ 5601.275298] Key type .llcrypt unregistered [ 5601.277047] Key type ._llcrypt unregistered [ 5629.474490] Key type ._llcrypt registered [ 5629.482441] Key type .llcrypt registered [ 5629.725749] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5631.869846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5632.019133] alg: No test for adler32 (adler32-zlib) [ 5633.563100] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 5634.070589] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5635.767175] Key type lgssc registered [ 5636.474752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5645.604631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5645.626659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.936981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.446954] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5657.061162] LustreError: 218350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.101781] LustreError: 218350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5662.188488] LustreError: 218351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5664.917324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.600481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5671.717387] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5681.104809] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5690.205711] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5701.677551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5702.144809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.658486] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5716.486157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5719.408774] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5730.397228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.515698] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5730.870393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.006754] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5750.284884] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5752.650586] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5764.660303] Lustre: Permanently deactivating lustre-OST0001 [ 5764.671871] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5774.548848] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5782.835950] Lustre: Permanently reactivating lustre-OST0001 [ 5782.859491] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5782.873532] Lustre: Skipped 2 previous similar messages [ 5789.292091] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.314362] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5789.327821] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5789.343157] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5789.457767] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5794.802276] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.807687] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5794.814813] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5794.820753] Lustre: Skipped 2 previous similar messages [ 5794.850970] Lustre: Skipped 1 previous similar message [ 5796.493794] Lustre: server umount lustre-OST0001 complete [ 5799.925867] LustreError: 220790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.939969] LustreError: 220790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5803.488263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5803.507176] LustreError: Skipped 1 previous similar message [ 5803.509088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5803.546930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.516162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5805.025242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5805.027337] LustreError: 220790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.047691] Lustre: Skipped 2 previous similar messages [ 5805.078913] LustreError: 220790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5806.811949] Lustre: server umount lustre-OST0000 complete [ 5814.240368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5814.245961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.259274] Lustre: Skipped 1 previous similar message [ 5814.268606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.009907] Lustre: server umount lustre-MDT0000 complete [ 5820.386149] LustreError: 218350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.403357] LustreError: 218350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5822.065891] LustreError: 218330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580389 with bad export cookie 11539244382563559425 [ 5822.076551] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.077583] LustreError: 218330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5822.407854] Lustre: server umount lustre-MDT0001 complete [ 5833.801584] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5836.531448] Key type lgssc unregistered [ 5836.905861] LNet: 223421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5836.911075] LNetError: 223421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5836.929719] LNet: Removed LNI 192.168.202.120@tcp [ 5837.941186] Key type .llcrypt unregistered [ 5837.942958] Key type ._llcrypt unregistered [ 5882.147706] Key type ._llcrypt registered [ 5882.150829] Key type .llcrypt registered [ 5882.261481] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5883.553471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5883.607094] alg: No test for adler32 (adler32-zlib) [ 5884.716505] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 5884.963638] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5886.641056] Key type lgssc registered [ 5888.479721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5896.726262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5896.738332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.924559] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5897.948723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5898.127180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.694634] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5913.312088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.368688] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5913.386394] Lustre: 225625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5913.566467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.795417] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5927.517699] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5934.822765] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5942.264137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5942.518125] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5942.777865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.676141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5950.463694] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5958.942971] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5962.091671] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5969.252240] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5977.149698] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5986.035292] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5999.548820] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5999.555659] Lustre: Skipped 1 previous similar message [ 6017.909954] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6017.915362] Lustre: Skipped 1 previous similar message [ 6051.078438] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6051.082389] Lustre: Skipped 3 previous similar messages [ 6116.645266] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6116.653119] Lustre: Skipped 7 previous similar messages [ 6148.576731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.586713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.595567] LustreError: Skipped 1 previous similar message [ 6148.603526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.631558] Lustre: Skipped 1 previous similar message [ 6148.644230] Lustre: Skipped 1 previous similar message [ 6149.890657] Lustre: server umount lustre-OST0000 complete [ 6158.819217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.842133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.851166] Lustre: Skipped 1 previous similar message [ 6160.907611] Lustre: server umount lustre-MDT0000 complete [ 6164.908427] LustreError: 225615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580732 with bad export cookie 15708453178586268246 [ 6164.918306] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.922319] LustreError: 225615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6165.214426] Lustre: server umount lustre-MDT0001 complete [ 6175.788458] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6179.020813] Key type lgssc unregistered [ 6179.363933] LNet: 230429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6179.369172] LNetError: 230429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6179.395442] LNet: Removed LNI 192.168.202.120@tcp [ 6180.281344] Key type .llcrypt unregistered [ 6180.285730] Key type ._llcrypt unregistered [ 6206.147092] Key type ._llcrypt registered [ 6206.150611] Key type .llcrypt registered [ 6206.293782] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6207.788255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6207.856752] alg: No test for adler32 (adler32-zlib) [ 6208.929212] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 6209.245689] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6210.975133] Key type lgssc registered [ 6212.323888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.199958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.265499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.800686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.778564] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6233.062525] LustreError: 231841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.085858] LustreError: 231841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6236.665725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.099352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.643522] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6251.341053] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6257.686781] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6265.799589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6266.328074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.388372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6274.186281] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6282.608643] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6298.087968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6298.100540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.115235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6302.179298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6302.188797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6302.191683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.356774] Lustre: server umount lustre-OST0000 complete [ 6312.416844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.433736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6312.452965] Lustre: Skipped 2 previous similar messages [ 6316.002764] Lustre: server umount lustre-MDT0000 complete [ 6317.540869] LustreError: 232967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.570138] LustreError: 232967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6320.751371] LustreError: 234425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580888 with bad export cookie 3462207574311057237 [ 6320.761564] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6320.793193] LustreError: 234425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6321.185427] Lustre: server umount lustre-MDT0001 complete [ 6335.811879] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6338.711128] Key type lgssc unregistered [ 6339.211579] LNet: 235547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6339.222717] LNetError: 235547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6340.265943] LNet: Removed LNI 192.168.202.120@tcp [ 6341.426511] Key type .llcrypt unregistered [ 6341.428510] Key type ._llcrypt unregistered [ 6368.513974] Key type ._llcrypt registered [ 6368.516935] Key type .llcrypt registered [ 6368.661432] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6369.838702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6369.908094] alg: No test for adler32 (adler32-zlib) [ 6371.218432] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 6371.553569] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6373.303220] Key type lgssc registered [ 6374.656119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6647.178992] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6650.595721] Key type lgssc unregistered [ 6651.087228] LNet: 237569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6651.098040] LNetError: 237569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6651.106843] LNet: Removed LNI 192.168.202.120@tcp [ 6652.003455] Key type .llcrypt unregistered [ 6652.004924] Key type ._llcrypt unregistered [ 6685.380764] Key type ._llcrypt registered [ 6685.383758] Key type .llcrypt registered [ 6685.583843] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 6705.562329] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6706.912570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6706.968997] alg: No test for adler32 (adler32-zlib) [ 6707.917067] Lustre: Lustre: Build Version: 2.17.58_3_g4a08d7b [ 6708.119195] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6709.847365] Key type lgssc registered [ 6711.294349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6748.537165] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6637 sec ========== 00:08:34 (1788581314) [ 6750.786038] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:08:36 (1788581316) === [ 6754.195398] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:08:40 (1788581320) === [ 6796.809728] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6799.476282] Key type lgssc unregistered [ 6799.734957] LNet: 242807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6799.738651] LNetError: 242807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6799.759614] LNet: Removed LNI 192.168.202.120@tcp [ 6800.686673] Key type .llcrypt unregistered [ 6800.691036] Key type ._llcrypt unregistered