[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 545746423 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003066] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.021371] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026531] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] *** VALIDATE tmpfs *** [ 0.030423] *** VALIDATE proc *** [ 0.032228] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041449] debug: unmapping init [mem 0xffffffff93859000-0xffffffff93860fff] [ 0.043843] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049007] ... max period: 00007fffffffffff [ 0.050006] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052246] rcu: Hierarchical SRCU implementation. [ 0.054345] smp: Bringing up secondary CPUs ... [ 0.056358] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.062580] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.097047] node 0 deferred pages initialised in 30ms [ 0.100009] devtmpfs: initialized [ 0.101143] x86/mm: Memory block size: 128MB [ 0.103949] gcov: version magic: 0x41383552 [ 0.105294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107290] pinctrl core: initialized pinctrl subsystem [ 0.108161] [ 0.108608] ************************************************************* [ 0.109017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110013] ** ** [ 0.111009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112011] ** ** [ 0.113015] ** This means that this kernel is built to expose internal ** [ 0.114015] ** IOMMU data structures, which may compromise security on ** [ 0.115015] ** your system. ** [ 0.116016] ** ** [ 0.117017] ** If you see this message and you are not debugging the ** [ 0.118017] ** kernel, report this immediately to your vendor! ** [ 0.119083] ** ** [ 0.120011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121008] ************************************************************* [ 0.122720] NET: Registered protocol family 16 [ 0.123497] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.126625] cpuidle: using governor menu [ 0.128400] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.130467] PCI: Using configuration type 1 for base access [ 0.132138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.141108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.144120] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.151179] cryptd: max_cpu_qlen set to 1000 [ 0.154000] ACPI: Added _OSI(Module Device) [ 0.155016] ACPI: Added _OSI(Processor Device) [ 0.157019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.160027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.169199] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182524] ACPI: Interpreter enabled [ 0.183105] ACPI: PM: (supports S0 S3 S4 S5) [ 0.184018] ACPI: Using IOAPIC for interrupt routing [ 0.185156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186922] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.203185] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211195] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216205] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.224725] acpiphp: Slot [2] registered [ 0.227482] acpiphp: Slot [5] registered [ 0.229260] acpiphp: Slot [6] registered [ 0.232170] acpiphp: Slot [7] registered [ 0.234389] acpiphp: Slot [8] registered [ 0.236235] acpiphp: Slot [9] registered [ 0.238629] acpiphp: Slot [10] registered [ 0.242127] acpiphp: Slot [3] registered [ 0.244775] acpiphp: Slot [4] registered [ 0.247539] acpiphp: Slot [11] registered [ 0.250576] acpiphp: Slot [12] registered [ 0.254378] acpiphp: Slot [13] registered [ 0.256345] acpiphp: Slot [14] registered [ 0.259377] acpiphp: Slot [15] registered [ 0.264945] acpiphp: Slot [16] registered [ 0.269191] acpiphp: Slot [17] registered [ 0.271416] acpiphp: Slot [18] registered [ 0.274169] acpiphp: Slot [19] registered [ 0.276353] acpiphp: Slot [20] registered [ 0.278147] acpiphp: Slot [21] registered [ 0.280481] acpiphp: Slot [22] registered [ 0.283252] acpiphp: Slot [23] registered [ 0.285270] acpiphp: Slot [24] registered [ 0.288276] acpiphp: Slot [25] registered [ 0.290490] acpiphp: Slot [26] registered [ 0.293409] acpiphp: Slot [27] registered [ 0.295294] acpiphp: Slot [28] registered [ 0.297509] acpiphp: Slot [29] registered [ 0.300230] acpiphp: Slot [30] registered [ 0.302285] acpiphp: Slot [31] registered [ 0.304517] PCI host bridge to bus 0000:00 [ 0.307039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310067] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.327423] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338208] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.353930] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377381] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382373] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386245] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390990] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.399026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.432023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.446606] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.458022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.468029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.532029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543575] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.576029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.589021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.612023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.635223] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.642033] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.651029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.672026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.686178] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.698025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.704025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.724096] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.737092] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.747021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.754023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.772025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.781000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.789577] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.796022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.813024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.823000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.824583] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.827330] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.829441] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.837031] iommu: Default domain type: Passthrough [ 0.839322] SCSI subsystem initialized [ 0.840145] ACPI: bus type USB registered [ 0.842119] usbcore: registered new interface driver usbfs [ 0.844091] usbcore: registered new interface driver hub [ 0.845098] usbcore: registered new device driver usb [ 0.847238] pps_core: LinuxPPS API ver. 1 registered [ 0.848011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.850237] PTP clock support registered [ 0.852452] EDAC MC: Ver: 3.0.0 [ 0.854587] PCI: Using ACPI for IRQ routing [ 0.855000] NetLabel: Initializing [ 0.855000] NetLabel: domain hash size = 128 [ 0.856014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.858103] NetLabel: unlabeled traffic allowed by default [ 0.860494] vgaarb: loaded [ 0.862061] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.864014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.871220] clocksource: Switched to clocksource kvm-clock [ 1.030393] VFS: Disk quotas dquot_6.6.0 [ 1.031754] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.035498] *** VALIDATE ramfs *** [ 1.037239] *** VALIDATE hugetlbfs *** [ 1.041854] pnp: PnP ACPI init [ 1.044993] pnp: PnP ACPI: found 6 devices [ 1.065855] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.069524] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.072073] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.074244] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.077139] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.079853] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.083114] NET: Registered protocol family 2 [ 1.086183] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.092075] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.096335] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.102182] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.107467] TCP: Hash tables configured (established 65536 bind 65536) [ 1.111930] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.118518] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.122653] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.127465] NET: Registered protocol family 1 [ 1.131881] RPC: Registered named UNIX socket transport module. [ 1.134660] RPC: Registered udp transport module. [ 1.137275] RPC: Registered tcp transport module. [ 1.139979] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.146538] NET: Registered protocol family 44 [ 1.148289] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.150974] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.154421] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.158526] PCI: CLS 0 bytes, default 64 [ 1.161279] Unpacking initramfs... [ 7.195628] debug: unmapping init [mem 0xffff8a3cbcc54000-0xffff8a3cbffbffff] [ 7.199707] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 7.201462] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 7.203882] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.332385] Initialise system trusted keyrings [ 8.337361] Key type blacklist registered [ 8.342819] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.368690] zbud: loaded [ 8.375831] *** VALIDATE nfs *** [ 8.377150] *** VALIDATE nfs4 *** [ 8.381702] pstore: using deflate compression [ 8.397941] Platform Keyring initialized [ 8.750851] NET: Registered protocol family 38 [ 8.752302] Key type asymmetric registered [ 8.753633] Asymmetric key parser 'x509' registered [ 8.762392] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.773271] io scheduler mq-deadline registered [ 8.777873] io scheduler kyber registered [ 8.784969] io scheduler bfq registered [ 8.797951] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.802895] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.815843] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.827101] ACPI: Power Button [PWRF] [ 8.934588] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.985177] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.020484] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.039237] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.073875] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.117338] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.152304] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.168380] Non-volatile memory driver v1.3 [ 9.172486] Linux agpgart interface v0.103 [ 9.228620] virtio_blk virtio1: [vda] 134136 512-byte logical blocks (68.7 MB/65.5 MiB) [ 9.236157] vda: detected capacity change from 0 to 68677632 [ 9.268803] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.271634] vdb: detected capacity change from 0 to 1073741824 [ 9.287723] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.290955] vdc: detected capacity change from 0 to 2621440000 [ 9.304836] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.307635] vdd: detected capacity change from 0 to 2621440000 [ 9.322389] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.325220] vde: detected capacity change from 0 to 4294967296 [ 9.354397] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.364168] vdf: detected capacity change from 0 to 4294967296 [ 9.376679] libphy: Fixed MDIO Bus: probed [ 9.389428] usbcore: registered new interface driver usbserial_generic [ 9.395685] usbserial: USB Serial support registered for generic [ 9.402818] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.418550] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.423923] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.427133] mousedev: PS/2 mouse device common for all mice [ 9.432108] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.439294] rtc_cmos 00:05: RTC can wake from S4 [ 9.444753] rtc_cmos 00:05: registered as rtc0 [ 9.455437] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.461866] intel_pstate: CPU model not supported [ 9.463853] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.476831] hid: raw HID events driver (C) Jiri Kosina [ 9.478859] usbcore: registered new interface driver usbhid [ 9.484734] usbhid: USB HID core driver [ 9.486127] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.487764] drop_monitor: Initializing network drop monitor service [ 9.487911] Initializing XFRM netlink socket [ 9.498807] NET: Registered protocol family 10 [ 9.523880] Segment Routing with IPv6 [ 9.525848] NET: Registered protocol family 17 [ 9.531063] mpls_gso: MPLS GSO support [ 9.536958] RAS: Correctable Errors collector initialized. [ 9.539091] AVX version of gcm_enc/dec engaged. [ 9.540597] AES CTR mode by8 optimization enabled [ 9.650679] sched_clock: Marking stable (9650649843, 0)->(10905897959, -1255248116) [ 9.656622] registered taskstats version 1 [ 9.661165] Loading compiled-in X.509 certificates [ 9.663196] zswap: loaded using pool lzo/zbud [ 9.719278] Key type big_key registered [ 9.763808] Key type encrypted registered [ 9.765873] ima: No TPM chip found, activating TPM-bypass! [ 9.768487] ima: Allocated hash algorithm: sha1 [ 9.771736] ima: No architecture policies found [ 9.773190] evm: Initialising EVM extended attributes: [ 9.775320] evm: security.selinux [ 9.776539] evm: security.ima [ 9.777794] evm: security.capability [ 9.780495] evm: HMAC attrs: 0x1 [ 9.790871] rtc_cmos 00:05: setting system clock to 2026-04-10 18:42:48 UTC (1775846568) [ 9.805576] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 9.828766] debug: unmapping init [mem 0xffffffff93582000-0xffffffff93858fff] [ 9.852248] Write protecting the kernel read-only data: 28672k [ 9.870850] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 9.877231] debug: unmapping init [mem 0xffffffff92514000-0xffffffff925fffff] [ 9.994231] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.021304] systemd[1]: Detected virtualization kvm. [ 10.023567] systemd[1]: Detected architecture x86-64. [ 10.027630] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.121722] systemd[1]: No hostname configured. [ 10.126249] systemd[1]: Set hostname to . [ 10.132657] random: systemd: uninitialized urandom read (16 bytes read) [ 10.140620] systemd[1]: Initializing machine ID from random generator. [ 10.205769] random: ln: uninitialized urandom read (6 bytes read) [ 10.568976] random: systemd: uninitialized urandom read (16 bytes read) [ 10.577851] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 10.625846] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 10.654910] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.492350] hrtimer: interrupt took 3343070 ns [ 12.724142] device-mapper: uevent: version 1.0.3 [ 12.727714] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 14.651068] virtio_net virtio0 ens2: renamed from eth0 [ 14.753905] random: fast init done [ 15.281562] scsi host0: ata_piix [ 15.323060] scsi host1: ata_piix [ 15.324493] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.338248] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 20.322785] random: crng init done [ 20.326688] random: 7 urandom warning(s) missed due to ratelimiting [ 23.888822] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 26.099157] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 29.606823] printk: systemd: 26 output lines suppressed due to ratelimiting [ 30.389221] SELinux: Disabled at runtime. [ 30.480687] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 30.494807] systemd[1]: Detected virtualization kvm. [ 30.498175] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.903162] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.913579] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.929663] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.938782] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.944902] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.967632] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.977979] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ 32.246900] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 33.938635] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.614972] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.679320] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.054436] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.156086] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 42.341224] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root supp[ 43.294233] NFS: Registering the id_resolver key type ort (11s / no limit)[ 43.298688] Key type id_resolver registered [ 43.326712] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg230-server login: [ 87.604914] libcfs: loading out-of-tree module taints kernel. [ 87.639427] Key type ._llcrypt registered [ 87.641445] Key type .llcrypt registered [ 87.731813] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 99.479356] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 100.435797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.445025] alg: No test for adler32 (adler32-zlib) [ 101.648206] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 102.079185] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 103.783573] Key type lgssc registered [ 104.707973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.189113] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.327680] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 156.403770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.435188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.666704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.692720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.760271] Lustre: lustre-MDT0000: new disk, initializing [ 157.831873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.846430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.634894] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.435897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.505236] Lustre: 6480:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.528374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.531474] Lustre: Skipped 1 previous similar message [ 171.594935] Lustre: lustre-MDT0001: new disk, initializing [ 171.648874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.667282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.671070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.681323] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.390690] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.959693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.194752] Lustre: lustre-OST0000: new disk, initializing [ 185.198797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.241921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.544842] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.470389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.478929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 190.528850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.107905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.179547] Lustre: lustre-OST0001: new disk, initializing [ 198.185919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 198.223024] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.284814] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.776631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.784501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.818199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.779053] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 217.128555] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.933398] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.607434] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing yml_node [ 232.493763] Lustre: DEBUG MARKER: Client: 2.17.52.1 [ 234.146358] Lustre: DEBUG MARKER: MDS: 2.17.52.1 [ 236.061885] Lustre: DEBUG MARKER: OSS: 2.17.52.1 [ 237.344509] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 10 14:46:35 EDT 2026 [ 249.968829] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 250.929775] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 264.160133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 264.167082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 264.179583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.183725] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.184588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.207169] Lustre: Skipped 3 previous similar messages [ 270.305612] Lustre: server umount lustre-MDT0000 complete [ 273.037578] LustreError: 7419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846831 with bad export cookie 4116274178633654394 [ 273.041842] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.043686] LustreError: 7419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 275.425412] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.425816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.428070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.435735] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 275.458793] Lustre: Skipped 3 previous similar messages [ 278.432329] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.432524] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 278.463126] Lustre: Skipped 2 previous similar messages [ 279.324365] Lustre: server umount lustre-MDT0001 complete [ 281.165769] Lustre: server umount lustre-OST0000 complete [ 283.213644] Lustre: server umount lustre-OST0001 complete [ 286.992351] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 292.085381] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 312.402337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.608866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 312.645178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 312.706828] Lustre: lustre-MDT0000: new disk, initializing [ 312.752411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.763384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 315.383534] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 322.011766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.092856] Lustre: 14944:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 322.132606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 322.136120] Lustre: Skipped 1 previous similar message [ 322.195770] Lustre: lustre-MDT0001: new disk, initializing [ 322.252686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 322.281207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 322.287696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 324.694221] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 329.701285] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 331.819209] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 335.737466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 337.489713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 337.586536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 339.369566] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 343.984486] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 346.621311] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 346.744747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 349.355838] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 349.527958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 353.252785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 353.254087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.257044] LustreError: Skipped 1 previous similar message [ 353.259151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.266380] Lustre: Skipped 1 previous similar message [ 353.273100] Lustre: Skipped 1 previous similar message [ 357.080995] Lustre: server umount lustre-OST0000 complete [ 363.489173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 363.489264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.490427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.504160] Lustre: Skipped 1 previous similar message [ 374.242518] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.342805] Lustre: server umount lustre-MDT0000 complete [ 376.529725] LustreError: 14936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846935 with bad export cookie 4116274178633655927 [ 376.530481] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 376.535412] LustreError: 14936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 376.677621] Lustre: server umount lustre-MDT0001 complete [ 378.594435] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:48:57 (1775846937) [ 382.203941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.458165] LustreError: 18810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.472853] LustreError: 18810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 382.515832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.519660] Lustre: Skipped 1 previous similar message [ 384.656776] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 387.553021] LustreError: 18811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.420156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.729873] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 394.576425] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 396.691757] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 400.303732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 403.424657] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 406.625696] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 420.321945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 420.322136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.325890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.329662] Lustre: Skipped 1 previous similar message [ 420.335166] Lustre: Skipped 6 previous similar messages [ 421.085930] Lustre: server umount lustre-OST0000 complete [ 425.443254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 429.435202] Lustre: server umount lustre-MDT0000 complete [ 430.560936] LustreError: 19870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.572148] LustreError: 19870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 431.617848] LustreError: 18791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846990 with bad export cookie 4116274178633656809 [ 431.619449] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 431.623376] LustreError: 18791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 431.848705] Lustre: server umount lustre-MDT0001 complete [ 437.707524] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 439.409435] Key type lgssc unregistered [ 439.590533] LNet: 22008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.594723] LNetError: 22008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.602583] LNet: Removed LNI 192.168.202.130@tcp [ 440.099333] Key type .llcrypt unregistered [ 440.101573] Key type ._llcrypt unregistered [ 452.292257] Key type ._llcrypt registered [ 452.293656] Key type .llcrypt registered [ 452.369797] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 452.918439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 452.993235] alg: No test for adler32 (adler32-zlib) [ 453.936809] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 454.068831] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 455.687316] Key type lgssc registered [ 456.304854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 460.867881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 460.893250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.222129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.941955] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 470.695562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.887672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 473.391953] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 477.042825] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.420074] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.790050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.980141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.082832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 487.458518] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 490.547132] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 501.733635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.734881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.741475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.755826] Lustre: Skipped 1 previous similar message [ 502.313318] Lustre: server umount lustre-OST0000 complete [ 506.849349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.850482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.861547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 506.865601] Lustre: Skipped 1 previous similar message [ 506.874056] Lustre: Skipped 2 previous similar messages [ 510.393819] Lustre: server umount lustre-MDT0000 complete [ 511.972858] LustreError: 23391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.984560] LustreError: 23391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 512.604085] LustreError: 25988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847071 with bad export cookie 1228147777300168630 [ 512.610781] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.615414] LustreError: 25988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.788352] Lustre: server umount lustre-MDT0001 complete [ 519.361531] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 521.086131] Key type lgssc unregistered [ 521.295459] LNet: 27007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.305121] LNetError: 27007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.324077] LNet: Removed LNI 192.168.202.130@tcp [ 521.859353] Key type .llcrypt unregistered [ 521.861096] Key type ._llcrypt unregistered [ 535.050580] Key type ._llcrypt registered [ 535.053516] Key type .llcrypt registered [ 535.111408] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 535.566016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 535.631734] alg: No test for adler32 (adler32-zlib) [ 536.600200] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 536.709206] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 538.303204] Key type lgssc registered [ 538.940578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 543.976967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 543.990543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.286062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.790177] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 553.221776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.431820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 555.593742] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 558.919538] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 561.155185] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.107358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.351878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.597790] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 576.713289] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.801686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 599.007969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 599.010792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.016796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.033325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 600.036468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.037994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.047054] Lustre: Skipped 1 previous similar message [ 601.835611] Lustre: server umount lustre-OST0000 complete [ 604.639925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 604.642855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.648318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.768106] Lustre: server umount lustre-MDT0000 complete [ 610.271930] LustreError: 28379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.279904] LustreError: 28379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 611.410721] LustreError: 28361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847170 with bad export cookie 15248221024905885413 [ 611.411920] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.416244] LustreError: 28361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 611.568737] Lustre: server umount lustre-MDT0001 complete [ 616.444251] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 617.885517] Key type lgssc unregistered [ 618.046740] LNet: 31928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.051936] LNetError: 31928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.061389] LNet: Removed LNI 192.168.202.130@tcp [ 618.570250] Key type .llcrypt unregistered [ 618.571401] Key type ._llcrypt unregistered [ 631.828347] Key type ._llcrypt registered [ 631.829953] Key type .llcrypt registered [ 631.887827] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 632.373560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 632.382686] alg: No test for adler32 (adler32-zlib) [ 633.282866] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 633.383793] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 634.991138] Key type lgssc registered [ 635.591038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.246385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 639.259366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.513433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.580922] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 647.255352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.445885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 649.542970] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 652.264836] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 654.233104] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 658.002339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 658.178869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.085807] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 664.005071] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 665.386678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 680.929748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 680.934917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.940509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 682.976865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 682.978362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 682.983105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.311429] Lustre: server umount lustre-OST0000 complete [ 688.097162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.097932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.101539] Lustre: Skipped 1 previous similar message [ 688.104189] Lustre: Skipped 1 previous similar message [ 693.216460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 693.218927] Lustre: Skipped 1 previous similar message [ 698.336263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 698.341899] Lustre: Skipped 1 previous similar message [ 700.384233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 700.471466] Lustre: server umount lustre-MDT0000 complete [ 702.433849] LustreError: 34012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.444171] LustreError: 34012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 702.560203] LustreError: 33282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847261 with bad export cookie 7501055717311691122 [ 702.563444] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.564919] LustreError: 33282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.760158] Lustre: server umount lustre-MDT0001 complete [ 707.954325] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 709.231636] Key type lgssc unregistered [ 709.379525] LNet: 36504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.386251] LNetError: 36504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.404812] LNet: Removed LNI 192.168.202.130@tcp [ 709.806203] Key type .llcrypt unregistered [ 709.807640] Key type ._llcrypt unregistered [ 721.656628] Key type ._llcrypt registered [ 721.658851] Key type .llcrypt registered [ 721.729703] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 722.168311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.262170] alg: No test for adler32 (adler32-zlib) [ 723.153691] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 723.253381] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 724.847220] Key type lgssc registered [ 725.367053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.232703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.244231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.490343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.779411] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 737.264403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.437874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 739.176932] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 741.984234] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.819482] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 746.981153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 747.157733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.791245] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 752.353721] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 755.366917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 763.839889] Lustre: lustre-OST0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 763.872926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 763.877270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.922085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 765.924933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.927454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.929872] Lustre: Skipped 1 previous similar message [ 767.312526] Lustre: server umount lustre-OST0000 complete [ 783.840218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 783.845473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.855727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.858303] Lustre: Skipped 1 previous similar message [ 786.914045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.924618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.177183] Lustre: server umount lustre-MDT0000 complete [ 790.446318] LustreError: 37858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847349 with bad export cookie 7739323444170302701 [ 790.451231] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.456536] LustreError: 37858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 796.763428] Lustre: server umount lustre-MDT0001 complete [ 802.247198] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 803.641565] Key type lgssc unregistered [ 803.813880] LNet: 41228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.822426] LNetError: 41228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.833243] LNet: Removed LNI 192.168.202.130@tcp [ 804.232404] Key type .llcrypt unregistered [ 804.234211] Key type ._llcrypt unregistered [ 816.968856] Key type ._llcrypt registered [ 816.970594] Key type .llcrypt registered [ 817.020377] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 817.581913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.593620] alg: No test for adler32 (adler32-zlib) [ 818.492293] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 818.614359] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 820.223275] Key type lgssc registered [ 820.684757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 824.263876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 824.275719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.481578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.501783] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 831.924148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.102539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.028133] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 837.072963] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 839.050646] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 842.432607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.623577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.234913] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 847.897720] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 850.856449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 862.176542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.179389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.183450] Lustre: Skipped 2 previous similar messages [ 862.186041] Lustre: Skipped 1 previous similar message [ 864.701782] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 867.269886] Lustre: server umount lustre-MDT0000 complete [ 867.297866] LustreError: 42601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.310345] LustreError: 42601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 869.231834] LustreError: 42580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847427 with bad export cookie 2669636279820310226 [ 869.238675] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.808912] LustreError: 43709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.832650] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 872.416109] LustreError: 42597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.417928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.433975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 874.463957] LustreError: 42596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 875.503767] Lustre: server umount lustre-MDT0001 complete [ 894.314922] Lustre: server umount lustre-OST0000 complete [ 901.526193] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 902.912330] Key type lgssc unregistered [ 903.079661] LNet: 46125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.086595] LNetError: 46125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.096675] LNet: Removed LNI 192.168.202.130@tcp [ 903.507952] Key type .llcrypt unregistered [ 903.509789] Key type ._llcrypt unregistered [ 917.061429] Key type ._llcrypt registered [ 917.062861] Key type .llcrypt registered [ 917.116348] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 917.586463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.611235] alg: No test for adler32 (adler32-zlib) [ 918.523668] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 918.648971] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 920.255520] Key type lgssc registered [ 920.850647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.858973] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.871318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.383286] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 945.503196] LustreError: 47472:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 972.127179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.715650] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 977.445882] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1145.562691] Lustre: server umount lustre-OST0000 complete [ 1161.998511] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1165.360248] Key type lgssc unregistered [ 1165.707666] LNet: 49028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.716153] LNetError: 49028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.757860] LNet: Removed LNI 192.168.202.130@tcp [ 1167.715659] Key type .llcrypt unregistered [ 1167.720747] Key type ._llcrypt unregistered [ 1194.807069] Key type ._llcrypt registered [ 1194.810287] Key type .llcrypt registered [ 1194.994039] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1196.001629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.155024] alg: No test for adler32 (adler32-zlib) [ 1197.199356] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1197.431159] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1199.207170] Key type lgssc registered [ 1200.214516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.891385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.915575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.420129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1212.822123] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1219.565075] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.579415] LustreError: 50426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1221.055483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.293751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1224.694776] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1230.916841] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1236.647552] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1242.865688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.218838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.272819] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1253.318205] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1254.535778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1256.928350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1256.938621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.958281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1260.004862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.011497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.779498] Lustre: server umount lustre-OST0000 complete [ 1265.121118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.125412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1265.134138] Lustre: Skipped 1 previous similar message [ 1265.146358] Lustre: Skipped 2 previous similar messages [ 1270.241318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.246704] Lustre: Skipped 1 previous similar message [ 1271.048560] Lustre: server umount lustre-MDT0000 complete [ 1274.709370] LustreError: 50407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847833 with bad export cookie 3701205005972005192 [ 1274.711652] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.724966] LustreError: 50407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.005300] Lustre: server umount lustre-MDT0001 complete [ 1286.222473] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1288.567898] Key type lgssc unregistered [ 1288.839670] LNet: 53603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.845914] LNetError: 53603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.858332] LNet: Removed LNI 192.168.202.130@tcp [ 1289.600582] Key type .llcrypt unregistered [ 1289.602571] Key type ._llcrypt unregistered [ 1310.419627] Key type ._llcrypt registered [ 1310.421579] Key type .llcrypt registered [ 1310.508912] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1311.664108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1311.700309] alg: No test for adler32 (adler32-zlib) [ 1312.677590] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1312.932708] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1314.591149] Key type lgssc registered [ 1315.459707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1322.496813] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1322.515868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1338.015364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1343.135602] LustreError: 54972:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 1369.761367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.044592] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1379.804707] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1387.310648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.702816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.755029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1391.909617] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1393.151425] LustreError: 54998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.180425] LustreError: 54998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1398.256985] LustreError: 54997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.112694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.469210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1404.370615] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1409.688761] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1413.372640] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1418.723139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.726399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.743972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.263935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.270792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.846862] Lustre: server umount lustre-OST0000 complete [ 1445.868578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.878697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.893313] Lustre: Skipped 1 previous similar message [ 1445.904561] Lustre: Skipped 2 previous similar messages [ 1450.977261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.990096] Lustre: Skipped 1 previous similar message [ 1456.096980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.102878] Lustre: Skipped 1 previous similar message [ 1458.143303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1458.326363] Lustre: server umount lustre-MDT0000 complete [ 1461.216818] LustreError: 55895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.223655] LustreError: 55895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1461.674429] LustreError: 54980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848020 with bad export cookie 1248102219815008732 [ 1461.677161] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.687505] LustreError: 54980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1468.138670] Lustre: server umount lustre-MDT0001 complete [ 1478.254464] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1480.534530] Key type lgssc unregistered [ 1480.849407] LNet: 58365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.856219] LNetError: 58365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.895561] LNet: Removed LNI 192.168.202.130@tcp [ 1482.740218] Key type .llcrypt unregistered [ 1482.745280] Key type ._llcrypt unregistered [ 1502.584981] Key type ._llcrypt registered [ 1502.586986] Key type .llcrypt registered [ 1502.675883] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1503.370709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.503811] alg: No test for adler32 (adler32-zlib) [ 1504.546730] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1504.767464] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1506.448042] Key type lgssc registered [ 1507.460706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.938277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1513.958886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.540471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.024629] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1525.399346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1525.658453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.977777] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1532.757123] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1535.553760] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1540.480811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.691898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.748093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1545.128692] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1549.038611] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1566.689936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.697511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.720218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1566.729341] Lustre: Skipped 1 previous similar message [ 1571.260889] Lustre: server umount lustre-OST0000 complete [ 1576.937237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.941567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1576.956969] Lustre: Skipped 1 previous similar message [ 1576.960335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.988273] Lustre: Skipped 1 previous similar message [ 1580.949319] Lustre: server umount lustre-MDT0000 complete [ 1582.052368] LustreError: 59735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.064701] LustreError: 59735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1584.233938] LustreError: 59720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848142 with bad export cookie 9957766306197162715 [ 1584.240399] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1584.241893] LustreError: 59720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1584.629176] Lustre: server umount lustre-MDT0001 complete [ 1600.730812] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1603.808775] Key type lgssc unregistered [ 1604.084337] LNet: 62942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.105234] LNetError: 62942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.167502] LNet: Removed LNI 192.168.202.130@tcp [ 1606.797560] Key type .llcrypt unregistered [ 1606.802485] Key type ._llcrypt unregistered [ 1638.886091] Key type ._llcrypt registered [ 1638.887605] Key type .llcrypt registered [ 1638.988415] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1639.876748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.951147] alg: No test for adler32 (adler32-zlib) [ 1640.957277] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1641.160625] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1642.823787] Key type lgssc registered [ 1643.746500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1651.523557] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 15:10:09 (1775848209) [ 1657.285496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.308301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.841352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.067117] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1669.095956] LustreError: 64609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.121904] LustreError: 64609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1670.426909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.844606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1675.166079] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1680.058282] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.384267] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1688.863404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.161643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.890094] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1698.990392] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1703.550269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1709.002144] Lustre: Failing over lustre-MDT0000 [ 1709.027974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.039216] Lustre: Skipped 1 previous similar message [ 1709.043972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.048210] Lustre: Skipped 1 previous similar message [ 1709.164562] Lustre: server umount lustre-MDT0000 complete [ 1711.052463] LustreError: 64606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.151280] LustreError: 64604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.180675] LustreError: 64604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1716.153701] LustreError: 64605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.459813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.549674] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.646442] Lustre: *** cfs_fail_loc=135, val=0*** [ 1716.722904] LustreError: 67360:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000 failed (-2) [ 1716.732178] LustreError: 67360:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.130@tcp: cfg command failed: rc = -2 [ 1716.737074] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1716.737074] [ 1716.741121] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1716.747322] LustreError: 67347:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1716.751524] LustreError: 67347:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 1716.759018] LustreError: 67347:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 1716.764932] LustreError: 67347:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 1716.785447] Lustre: server umount lustre-MDT0000 complete [ 1716.787229] LustreError: 67347:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1721.273783] LustreError: 64604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.289219] LustreError: 64604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1722.228313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.364631] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1722.568392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.600358] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1725.583947] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1726.382311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1727.985617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1728.015144] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1728.055891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1731.556191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1731.564082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.583128] Lustre: Skipped 1 previous similar message [ 1731.599506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.088168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.092788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.478946] Lustre: server umount lustre-OST0000 complete [ 1743.332572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1743.334213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.342622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.351586] Lustre: Skipped 1 previous similar message [ 1743.353866] Lustre: Skipped 1 previous similar message [ 1748.374282] Lustre: server umount lustre-MDT0000 complete [ 1748.447872] LustreError: 64610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.457231] LustreError: 64610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1750.753889] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848309 with bad export cookie 17845340357425596840 [ 1750.756421] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1750.760073] LustreError: 64591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1750.974481] Lustre: server umount lustre-MDT0001 complete [ 1758.295914] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1760.123332] Key type lgssc unregistered [ 1760.289568] LNet: 69109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.297272] LNetError: 69109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.307577] LNet: Removed LNI 192.168.202.130@tcp [ 1760.769681] Key type .llcrypt unregistered [ 1760.771785] Key type ._llcrypt unregistered [ 1774.319120] Key type ._llcrypt registered [ 1774.323816] Key type .llcrypt registered [ 1774.370470] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1774.859559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1774.892266] alg: No test for adler32 (adler32-zlib) [ 1775.843840] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1775.962690] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1777.567108] Key type lgssc registered [ 1778.110839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.674098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.682198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.954757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.451116] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1793.506198] LustreError: 70485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.561972] LustreError: 70485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1796.159979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.761266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1805.038904] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1815.442634] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1822.367788] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1832.890244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1833.480548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.367559] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1850.335741] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1854.059346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1865.099447] Lustre: Failing over lustre-MDT0000 [ 1865.697583] Lustre: server umount lustre-MDT0000 complete [ 1867.222749] LustreError: 70481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.236570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.256379] LustreError: 70481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1867.277915] Lustre: Skipped 1 previous similar message [ 1872.307176] LustreError: 72571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.330460] LustreError: 72571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1876.823992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.949767] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.113742] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1877.205293] LustreError: 73252:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000 failed (-9) [ 1877.208339] LustreError: 73252:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.130@tcp: cfg command failed: rc = -9 [ 1877.212694] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1877.212694] [ 1877.217371] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1877.223642] LustreError: 73240:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1877.227950] LustreError: 73240:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -9 [ 1877.231988] LustreError: 73240:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 1877.236930] LustreError: 73240:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 1877.256395] Lustre: server umount lustre-MDT0000 complete [ 1877.259362] LustreError: 73240:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1877.427744] LustreError: 70480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.454374] LustreError: 70480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1882.553582] LustreError: 70481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.584516] LustreError: 70481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1884.964180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.079420] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.402384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.455839] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1887.600753] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1890.089707] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1890.814897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1890.846271] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1890.910242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1901.032205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.044415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.058753] Lustre: Skipped 2 previous similar messages [ 1901.079498] Lustre: Skipped 1 previous similar message [ 1906.146410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.155815] Lustre: Skipped 1 previous similar message [ 1911.267479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.275110] Lustre: Skipped 1 previous similar message [ 1913.311283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1913.479898] Lustre: server umount lustre-OST0000 complete [ 1921.504260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1921.506257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.510405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.524712] Lustre: Skipped 1 previous similar message [ 1923.734574] Lustre: server umount lustre-MDT0000 complete [ 1926.624072] LustreError: 70484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.644751] LustreError: 70484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1927.317816] LustreError: 70465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848486 with bad export cookie 11506567487789465081 [ 1927.333344] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.344777] LustreError: 70465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1927.536421] Lustre: server umount lustre-MDT0001 complete [ 1937.980068] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1940.952802] Key type lgssc unregistered [ 1941.283634] LNet: 75008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.291861] LNetError: 75008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.305801] LNet: Removed LNI 192.168.202.130@tcp [ 1942.292249] Key type .llcrypt unregistered [ 1942.294487] Key type ._llcrypt unregistered [ 1955.685709] Key type ._llcrypt registered [ 1955.690717] Key type .llcrypt registered [ 1955.810501] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1956.092437] Key type .llcrypt unregistered [ 1956.095384] Key type ._llcrypt unregistered [ 1986.584642] Key type ._llcrypt registered [ 1986.587284] Key type .llcrypt registered [ 1986.711834] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1988.470671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.551505] alg: No test for adler32 (adler32-zlib) [ 1989.676739] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1990.101662] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1991.871338] Key type lgssc registered [ 1993.119560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.706705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2001.728890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.274102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.416147] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2012.648409] LustreError: 76879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.663347] LustreError: 76879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2017.761061] LustreError: 76880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.438436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.825613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.494461] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2030.552445] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2035.358971] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2042.607986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.040022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.208054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2049.784815] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2056.413274] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2068.960257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2068.971400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.984566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2068.995329] Lustre: Skipped 1 previous similar message [ 2073.337470] Lustre: server umount lustre-OST0000 complete [ 2079.202125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.224197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.233350] Lustre: Skipped 2 previous similar messages [ 2084.015988] Lustre: server umount lustre-MDT0000 complete [ 2084.320483] LustreError: 76875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.329206] LustreError: 76875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2087.503847] LustreError: 76860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848646 with bad export cookie 2823129937717277296 [ 2087.509529] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.520510] LustreError: 76860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2087.725044] Lustre: server umount lustre-MDT0001 complete [ 2098.267027] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2101.838802] Key type lgssc unregistered [ 2102.305954] LNet: 80059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.309523] LNetError: 80059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.327631] LNet: Removed LNI 192.168.202.130@tcp [ 2103.572439] Key type .llcrypt unregistered [ 2103.581144] Key type ._llcrypt unregistered [ 2135.169545] Key type ._llcrypt registered [ 2135.177461] Key type .llcrypt registered [ 2135.282535] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2137.026080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.107990] alg: No test for adler32 (adler32-zlib) [ 2138.293346] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2138.628296] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2140.399166] Key type lgssc registered [ 2141.974047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.257294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.282968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.893746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.700225] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2162.149124] LustreError: 81456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.176209] LustreError: 81456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2167.155797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.264617] LustreError: 81455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.657949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2172.743247] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2180.204401] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2184.575516] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2191.261827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2191.729382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.956458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2197.717536] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2203.372271] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2212.330238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.346361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.405312] LustreError: 83053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.418792] LustreError: 83053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2212.567446] Lustre: server umount lustre-OST0000 complete [ 2217.445994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.456872] Lustre: Skipped 1 previous similar message [ 2217.466373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.478717] Lustre: Skipped 1 previous similar message [ 2222.565585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.680816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.701805] Lustre: Skipped 2 previous similar messages [ 2231.263232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2231.387957] Lustre: server umount lustre-MDT0000 complete [ 2232.822884] LustreError: 81456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.845087] LustreError: 81456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2236.707459] LustreError: 83073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848795 with bad export cookie 15627859890237375232 [ 2236.717364] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.735460] LustreError: 83073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2237.058681] Lustre: server umount lustre-MDT0001 complete [ 2250.262601] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2253.940140] Key type lgssc unregistered [ 2254.183888] LNet: 84638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.212657] LNetError: 84638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.268896] LNet: Removed LNI 192.168.202.130@tcp [ 2256.054230] Key type .llcrypt unregistered [ 2256.059379] Key type ._llcrypt unregistered [ 2285.021476] Key type ._llcrypt registered [ 2285.024070] Key type .llcrypt registered [ 2285.173435] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2286.327162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.397226] alg: No test for adler32 (adler32-zlib) [ 2287.775816] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2288.066915] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2289.791465] Key type lgssc registered [ 2291.721978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2300.773938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2300.807029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.340446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.343792] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2312.678154] LustreError: 86035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.686914] LustreError: 86035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2315.409367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.101146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2320.965646] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2327.342463] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2332.209748] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2339.513162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.933164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.199589] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2348.161437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2352.108982] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2366.944264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2366.951085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.984897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.992429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.996042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2369.004188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.586882] Lustre: server umount lustre-OST0000 complete [ 2377.699072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2377.713121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.761473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.764386] Lustre: Skipped 1 previous similar message [ 2382.139215] Lustre: server umount lustre-MDT0000 complete [ 2384.352112] LustreError: 86036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.373128] LustreError: 86036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2387.194113] LustreError: 88196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848945 with bad export cookie 6965229766256804886 [ 2387.206738] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.464620] Lustre: server umount lustre-MDT0001 complete [ 2399.306490] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2402.323790] Key type lgssc unregistered [ 2402.581709] LNet: 89245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.603206] LNetError: 89245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2402.621925] LNet: Removed LNI 192.168.202.130@tcp [ 2403.563720] Key type .llcrypt unregistered [ 2403.565936] Key type ._llcrypt unregistered [ 2429.047604] Key type ._llcrypt registered [ 2429.049439] Key type .llcrypt registered [ 2429.115302] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2430.270444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2430.473250] alg: No test for adler32 (adler32-zlib) [ 2431.602779] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2431.839828] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2433.511346] Key type lgssc registered [ 2434.317134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2441.339666] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2441.355543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.927384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2462.047225] LustreError: 90615:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2488.736652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.729760] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2499.301919] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2505.043406] Lustre: server umount lustre-OST0000 complete [ 2513.919500] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:24:31 (1775849071) [ 2521.563812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.183593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.895687] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2527.208502] LustreError: 92140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.224649] LustreError: 92140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2531.300567] LustreError: 92141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.419497] LustreError: 92140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.973317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.551938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.294974] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2550.078796] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.624855] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2560.572187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.973944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.175874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2568.562616] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2575.069502] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2587.617687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.623734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2587.635967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.773103] Lustre: server umount lustre-OST0000 complete [ 2597.858991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.867892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.882265] Lustre: Skipped 2 previous similar messages [ 2597.922562] Lustre: Skipped 2 previous similar messages [ 2602.978937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.991468] Lustre: Skipped 1 previous similar message [ 2608.097765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.114869] Lustre: Skipped 1 previous similar message [ 2611.167196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2611.469485] Lustre: server umount lustre-MDT0000 complete [ 2613.216923] LustreError: 92140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.258597] LustreError: 92140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2615.584265] LustreError: 92122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849174 with bad export cookie 3545683510024292195 [ 2615.609625] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.107780] Lustre: server umount lustre-MDT0001 complete [ 2627.258568] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2631.492191] Key type lgssc unregistered [ 2631.901170] LNet: 95481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.917229] LNetError: 95481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2631.931326] LNet: Removed LNI 192.168.202.130@tcp [ 2632.752976] Key type .llcrypt unregistered [ 2632.760458] Key type ._llcrypt unregistered [ 2656.313843] Key type ._llcrypt registered [ 2656.316250] Key type .llcrypt registered [ 2656.416377] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2657.689698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2657.713135] alg: No test for adler32 (adler32-zlib) [ 2659.230774] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2659.548170] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2661.216335] Key type lgssc registered [ 2662.347743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2670.209726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2670.229380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.867872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2676.346362] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2681.321709] LustreError: 96893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.352809] LustreError: 96893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2686.434360] LustreError: 96892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.700383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.106747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.963646] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2698.347750] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2703.469507] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2713.142672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.926211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.971895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2721.240207] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2727.712837] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2741.732789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.733535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.756525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.768701] Lustre: Skipped 1 previous similar message [ 2741.784061] Lustre: Skipped 1 previous similar message [ 2746.853916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2746.876613] Lustre: Skipped 1 previous similar message [ 2747.689772] Lustre: server umount lustre-OST0000 complete [ 2751.969722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.984173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.996083] Lustre: Skipped 1 previous similar message [ 2757.092437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.099078] Lustre: Skipped 1 previous similar message [ 2757.770039] Lustre: server umount lustre-MDT0000 complete [ 2761.875156] LustreError: 96874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849320 with bad export cookie 2090062086897123052 [ 2761.886907] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.207329] Lustre: server umount lustre-MDT0001 complete [ 2774.443786] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2778.064416] Key type lgssc unregistered [ 2778.382663] LNet: 100096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.394467] LNetError: 100096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.432626] LNet: Removed LNI 192.168.202.130@tcp [ 2780.438600] Key type .llcrypt unregistered [ 2780.441715] Key type ._llcrypt unregistered [ 2799.854527] Key type ._llcrypt registered [ 2799.856818] Key type .llcrypt registered [ 2799.972771] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2801.019540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2801.069440] alg: No test for adler32 (adler32-zlib) [ 2802.107638] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 2802.309797] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2803.935189] Key type lgssc registered [ 2805.096358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2813.008031] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2813.043419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.578991] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2833.695305] LustreError: 101323:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2860.447209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.980801] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2872.695830] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2879.658945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.807046] LustreError: 100659:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000906afab7 x1862113019562240/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2880.060608] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2880.076270] LustreError: 102234:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2880.080744] LustreError: 102234:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 2880.090536] LustreError: 102234:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 2880.195470] Lustre: server umount lustre-MDT0000 complete [ 2880.197763] LustreError: 102234:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2900.447903] Lustre: 100660:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775849443/real 1775849443] req@00000000feefbb6f x1862113019563392/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775849459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2900.478573] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.504325] LustreError: 101335:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2921.066785] Lustre: server umount lustre-OST0000 complete [ 2932.383483] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2945.003650] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2986.132591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.716472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2986.835398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2986.975359] Lustre: lustre-MDT0000: new disk, initializing [ 2987.131603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.170114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2992.289693] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3003.703749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.845989] Lustre: 104995:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3003.911637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3003.917379] Lustre: Skipped 1 previous similar message [ 3003.982705] Lustre: lustre-MDT0001: new disk, initializing [ 3004.111216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.160475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3004.175049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3009.878205] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3018.434639] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3022.681877] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3029.317158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.521563] Lustre: lustre-OST0000: new disk, initializing [ 3029.525783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3029.616535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.235511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3036.253238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3036.272884] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3036.381966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3042.999382] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3047.801241] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3047.997404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3052.154708] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3052.423773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3057.123585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3057.125537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.128281] LustreError: Skipped 1 previous similar message [ 3057.132132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3057.147905] Lustre: Skipped 1 previous similar message [ 3057.169035] Lustre: Skipped 1 previous similar message [ 3062.242222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3062.258440] Lustre: Skipped 1 previous similar message [ 3067.372796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.388748] Lustre: Skipped 1 previous similar message [ 3068.383173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3068.568489] Lustre: server umount lustre-OST0000 complete [ 3072.487602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.500429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.509336] Lustre: Skipped 1 previous similar message [ 3072.533608] Lustre: Skipped 1 previous similar message [ 3077.600828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3077.608200] Lustre: Skipped 1 previous similar message [ 3080.483577] Lustre: server umount lustre-MDT0000 complete [ 3084.434520] LustreError: 104989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849643 with bad export cookie 11664219751641514944 [ 3084.451302] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.458591] LustreError: 104989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3084.963640] Lustre: server umount lustre-MDT0001 complete [ 3092.799386] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:34:10 (1775849650) [ 3096.088590] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3112.968349] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 3122.367647] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3164.231439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.483835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3164.525235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3164.638778] Lustre: lustre-MDT0000: new disk, initializing [ 3164.786419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.819542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3168.686268] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3179.253155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.367145] Lustre: 111470:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.413916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3179.417700] Lustre: Skipped 1 previous similar message [ 3179.479929] Lustre: lustre-MDT0001: new disk, initializing [ 3179.571558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3179.598741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3179.622940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3183.735784] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3192.514966] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3196.784423] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3204.270618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3204.506741] Lustre: lustre-OST0000: new disk, initializing [ 3204.511066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3204.582342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.278534] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3214.951272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3214.967828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3215.072958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3220.010534] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3225.698982] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3226.011918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3230.309296] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3230.514502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3235.828191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.828992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.837042] LustreError: Skipped 1 previous similar message [ 3235.842673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.851394] Lustre: Skipped 1 previous similar message [ 3235.873756] Lustre: Skipped 1 previous similar message [ 3238.627074] Lustre: server umount lustre-OST0000 complete [ 3246.050987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.054035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3246.062776] Lustre: Skipped 1 previous similar message [ 3248.728618] Lustre: server umount lustre-MDT0000 complete [ 3251.178873] LustreError: 111479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.213777] LustreError: 111479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3252.761639] LustreError: 111464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849811 with bad export cookie 11664219751641515833 [ 3252.790762] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.145985] Lustre: server umount lustre-MDT0001 complete [ 3262.190663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.588864] LustreError: 115280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.696684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.528328] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3268.067069] LustreError: 115281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.199682] LustreError: 115280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.366491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.927834] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3288.313486] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3294.252976] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3301.348496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.547697] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3314.633344] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3321.980187] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3328.483316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.492644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.520786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.526244] Lustre: Skipped 2 previous similar messages [ 3332.709331] Lustre: server umount lustre-OST0000 complete [ 3338.210018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3338.216085] LustreError: Skipped 1 previous similar message [ 3338.229915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.243962] Lustre: Skipped 1 previous similar message [ 3342.985882] Lustre: server umount lustre-MDT0000 complete [ 3343.842450] LustreError: 115277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.859033] LustreError: 115277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3346.899148] LustreError: 115262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849905 with bad export cookie 11664219751641516715 [ 3346.911919] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3346.913778] LustreError: 115262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3347.331243] Lustre: server umount lustre-MDT0001 complete [ 3357.971659] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3360.602454] Key type lgssc unregistered [ 3360.875838] LNet: 118559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3360.882923] LNetError: 118559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3361.901627] LNet: Removed LNI 192.168.202.130@tcp [ 3362.741265] Key type .llcrypt unregistered [ 3362.743217] Key type ._llcrypt unregistered [ 3377.840396] Key type ._llcrypt registered [ 3377.842755] Key type .llcrypt registered [ 3377.975398] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 3390.512525] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3391.302157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3391.442021] alg: No test for adler32 (adler32-zlib) [ 3392.483061] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 3392.679345] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3394.383596] Key type lgssc registered [ 3395.296837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3435.268954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3435.298992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.663185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3436.686788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3436.808280] Lustre: lustre-MDT0000: new disk, initializing [ 3436.933118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.959850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3441.160804] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3451.684726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.882773] Lustre: 121656:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3451.984676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3451.987982] Lustre: Skipped 1 previous similar message [ 3452.076300] Lustre: lustre-MDT0001: new disk, initializing [ 3452.165618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3452.198659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3452.205625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3457.137128] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3465.814293] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3469.980200] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3476.207766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.513594] Lustre: lustre-OST0000: new disk, initializing [ 3476.520465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3476.613530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.239171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3482.253553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3482.463350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3483.125569] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3490.652162] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3495.638302] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3495.817833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3500.145494] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3500.382904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3503.082693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.084362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.092626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.099785] Lustre: Skipped 1 previous similar message [ 3508.222910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.233580] Lustre: Skipped 2 previous similar messages [ 3508.573968] Lustre: server umount lustre-OST0000 complete [ 3513.311836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.324679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.340898] Lustre: Skipped 1 previous similar message [ 3513.347212] Lustre: Skipped 1 previous similar message [ 3518.441718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.446401] Lustre: Skipped 1 previous similar message [ 3519.384425] Lustre: server umount lustre-MDT0000 complete [ 3523.015770] LustreError: 121647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850081 with bad export cookie 15680234277728058381 [ 3523.021834] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.027541] LustreError: 121647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3523.523276] Lustre: server umount lustre-MDT0001 complete [ 3532.413288] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:41:30 (1775850090) [ 3540.244366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.686743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.027227] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3546.090563] LustreError: 125666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3546.122384] LustreError: 125666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3551.218436] LustreError: 125665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.199863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.578389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.065216] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3565.482600] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3569.715849] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3575.265525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3575.286969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.304272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3578.681407] Lustre: server umount lustre-MDT0000 complete [ 3579.363423] LustreError: 125665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.380566] LustreError: 125665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3582.518573] LustreError: 125645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850141 with bad export cookie 15680234277728059263 [ 3582.528027] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3582.880265] Lustre: server umount lustre-MDT0001 complete [ 3592.029671] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:42:29 (1775850149) [ 3600.348311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3615.968331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3621.087592] LustreError: 127853:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 3646.816809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.865163] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3658.764335] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3661.062789] Lustre: server umount lustre-OST0000 complete [ 3669.516334] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:43:47 (1775850227) [ 3676.929611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.329151] LustreError: 129187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.399657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.686790] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3682.793141] LustreError: 129188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.774559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.622412] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3701.416762] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3705.731639] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3712.543269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3712.992091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.013841] Lustre: Skipped 1 previous similar message [ 3720.087259] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3726.178720] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3737.056903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3737.062245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.074452] Lustre: Skipped 1 previous similar message [ 3737.080423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.085055] Lustre: Skipped 2 previous similar messages [ 3740.624367] Lustre: server umount lustre-MDT0000 complete [ 3743.713660] LustreError: 129187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.728903] LustreError: 129187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3745.158180] LustreError: 129169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850303 with bad export cookie 15680234277728059921 [ 3745.160070] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3745.168046] LustreError: 129169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3745.548702] Lustre: server umount lustre-MDT0001 complete [ 3761.085341] Lustre: server umount lustre-OST0000 complete [ 3770.835162] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:45:28 (1775850328) [ 3778.856932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.398435] LustreError: 132506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.464023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.105632] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3792.910221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.830850] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3804.403124] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.745973] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3818.069337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.437683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3825.757280] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3832.415360] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3837.631878] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3837.935741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3843.228803] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3843.454494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3850.210276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.213375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.230049] LustreError: Skipped 1 previous similar message [ 3850.248893] Lustre: Skipped 2 previous similar messages [ 3850.267154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.279291] Lustre: Skipped 4 previous similar messages [ 3852.091199] Lustre: server umount lustre-OST0000 complete [ 3859.941967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3859.949444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.979611] Lustre: Skipped 1 previous similar message [ 3862.924670] Lustre: server umount lustre-MDT0000 complete [ 3865.568468] LustreError: 132501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.585315] LustreError: 132501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3866.905174] LustreError: 132488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850425 with bad export cookie 15680234277728060999 [ 3866.908279] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3866.920987] LustreError: 132488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3867.238707] Lustre: server umount lustre-MDT0001 complete [ 3876.888348] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:47:14 (1775850434) [ 3884.809624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3900.448986] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3905.567789] LustreError: 136294:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 3931.295384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.298828] Lustre: Skipped 2 previous similar messages [ 3937.808418] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3943.900468] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3952.258146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.635405] LustreError: 136319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.799645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3957.537961] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3966.636638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.637994] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3978.507834] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.151843] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3988.891893] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3989.351309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3994.830154] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3995.141454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3997.682047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3997.698512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.721077] Lustre: Skipped 1 previous similar message [ 3997.736804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.749871] Lustre: Skipped 3 previous similar messages [ 4003.650610] Lustre: server umount lustre-MDT0000 complete [ 4008.262092] LustreError: 136301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850566 with bad export cookie 15680234277728062021 [ 4008.275672] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.774874] Lustre: server umount lustre-MDT0001 complete [ 4023.844718] Lustre: server umount lustre-OST0000 complete [ 4035.914774] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:49:52 (1775850592) [ 4046.172055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4061.919657] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4067.039703] LustreError: 140123:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 4099.423839] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4105.521313] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4113.218078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.363650] LustreError: 119878:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000333c1f1c x1862113638370176/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4113.624531] LustreError: 140148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.647634] LustreError: 140148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4113.827482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4118.316630] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4130.308744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.469458] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4142.268536] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4146.402083] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4153.663715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.828994] Lustre: lustre-OST0001: new disk, initializing [ 4153.832993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4159.933165] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4160.070183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4160.087954] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4160.222148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4168.380157] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4173.594698] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4173.839986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4178.750709] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4178.972739] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4181.988105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.003555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.027981] Lustre: Skipped 2 previous similar messages [ 4182.053421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.062239] Lustre: Skipped 3 previous similar messages [ 4187.654797] Lustre: server umount lustre-OST0000 complete [ 4206.047121] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4206.301260] Lustre: server umount lustre-OST0001 complete [ 4216.245293] Lustre: server umount lustre-MDT0000 complete [ 4220.069723] LustreError: 142659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850778 with bad export cookie 15680234277728062973 [ 4220.085316] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4220.088517] LustreError: 142659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4220.570422] Lustre: server umount lustre-MDT0001 complete [ 4246.143115] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:53:23 (1775850803) [ 4248.212596] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4250.118317] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4251.937541] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:53:29 (1775850809) [ 4259.191708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.482085] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4259.509224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4259.753777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.761491] Lustre: Skipped 6 previous similar messages [ 4265.425146] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4275.125091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.242902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4275.269995] Lustre: 146072:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4281.085203] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4288.267451] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4294.277690] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4302.336054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.586653] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4309.631834] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4311.759716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4316.392414] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4322.071762] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4322.390552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4327.456297] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4327.701330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4332.010106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4332.033531] LustreError: Skipped 3 previous similar messages [ 4332.043984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.051254] Lustre: Skipped 5 previous similar messages [ 4332.055410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4332.057930] Lustre: Skipped 9 previous similar messages [ 4344.287232] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4344.592574] Lustre: server umount lustre-OST0000 complete [ 4369.698843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.096063] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4373.111509] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4373.125562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4377.698280] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4384.110722] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4391.305708] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4391.592281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.961398] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4397.171858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4402.007908] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ee4c5fe3000.ost_server_uuid 50 [ 4403.921287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ee4c5fe3000.ost_server_uuid in FULL state after 0 sec [ 4404.141788] LustreError: 146080:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.30@tcp) returned error from blocking AST (req@00000000135c44fc x1862113638479488 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000cfd1cee3/0xd99b61ff86e29757 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.30@tcp remote: 0xf3ce29cdc819c494 expref: 5 pid: 147278 timeout: 4504 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4404.203156] LustreError: lustre-MDT0000: A client on nid 192.168.202.30@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4404.228868] LustreError: 146069:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.30@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000cfd1cee3/0xd99b61ff86e29757 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.30@tcp remote: 0xf3ce29cdc819c494 expref: 6 pid: 147278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4409.314785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4409.333768] LustreError: Skipped 1 previous similar message [ 4415.723138] Lustre: server umount lustre-OST0000 complete [ 4428.461577] Lustre: server umount lustre-MDT0000 complete [ 4429.801635] LustreError: 146084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.840081] LustreError: 146084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 4432.556630] LustreError: 146066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775850991 with bad export cookie 15680234277728064156 [ 4432.560924] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4432.569306] LustreError: 146066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4439.104964] Lustre: server umount lustre-MDT0001 complete [ 4452.638637] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4457.090981] Key type lgssc unregistered [ 4457.500229] LNet: 151569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4457.506272] LNetError: 151569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4457.524680] LNet: Removed LNI 192.168.202.130@tcp [ 4458.740726] Key type .llcrypt unregistered [ 4458.743755] Key type ._llcrypt unregistered [ 4486.855171] Key type ._llcrypt registered [ 4486.857540] Key type .llcrypt registered [ 4486.962216] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4488.032217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4488.124726] alg: No test for adler32 (adler32-zlib) [ 4489.332290] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 4489.566445] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4491.303422] Key type lgssc registered [ 4492.854810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4501.141457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4501.169983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.678402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.246633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4513.251470] LustreError: 152965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.280428] LustreError: 152965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4515.744812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.140693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4520.418311] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4526.665691] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4531.514179] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4539.499216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.181365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4547.375814] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4547.378669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4554.134095] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4560.086709] Lustre: Failing over lustre-MDT0000 [ 4560.373175] Lustre: server umount lustre-MDT0000 complete [ 4560.863840] LustreError: 154483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.401269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4562.411384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.425452] LustreError: 152965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.914152] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.932551] Lustre: Skipped 1 previous similar message [ 4568.034677] LustreError: 152964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.079687] LustreError: 152964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4572.855339] LustreError: 154581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.30@tcp arrived at 1775851131 with bad export cookie 16435695080272808815 [ 4573.154507] LustreError: 152964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.187190] LustreError: 152964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4579.295137] Lustre: 152276:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775851121/real 1775851121] req@00000000b65cecf9 x1862114788534528/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775851137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4579.355988] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.340984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.423518] LustreError: 152960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.454511] LustreError: 152960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4589.535793] LustreError: 152273:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000544f00b8 x1862114788539776/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4590.272754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.337593] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4595.682687] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4595.690251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4596.835644] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4600.395508] Lustre: lustre-MDT0000: Denying connection for new client f51ea3b2-4e9c-4b47-be2d-bb0ff26057f8 (at 192.168.202.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 4611.040934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.046800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.061638] Lustre: Skipped 1 previous similar message [ 4611.079529] Lustre: Skipped 1 previous similar message [ 4615.657262] Lustre: server umount lustre-OST0000 complete [ 4620.865807] LustreError: 156349:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4620.870106] Lustre: 155711:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4620.876086] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4620.890097] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4620.901287] LustreError: 155711:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006bb4bce5 x1862114788556160/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4620.919528] LustreError: 155711:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4620.926715] LustreError: 155711:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4620.936596] Lustre: lustre-MDT0000: Recovery over after 0:25, of 2 clients 0 recovered and 2 were evicted. [ 4621.224948] Lustre: server umount lustre-MDT0000 complete [ 4621.280959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.305465] LustreError: 152961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.326849] LustreError: 152961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4624.706842] LustreError: 152945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851183 with bad export cookie 16435695080272808913 [ 4624.711416] LustreError: 152273:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4624.715598] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.717870] LustreError: 152945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4631.164237] Lustre: server umount lustre-MDT0001 complete [ 4641.322630] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4644.011543] Key type lgssc unregistered [ 4644.296728] LNet: 157084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4644.306596] LNetError: 157084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4645.350946] LNet: Removed LNI 192.168.202.130@tcp [ 4646.126743] Key type .llcrypt unregistered [ 4646.129521] Key type ._llcrypt unregistered [ 4674.800588] Key type ._llcrypt registered [ 4674.804128] Key type .llcrypt registered [ 4674.903687] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4676.289122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4676.476026] alg: No test for adler32 (adler32-zlib) [ 4677.518099] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 4677.697456] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4679.367173] Key type lgssc registered [ 4680.631333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4690.299090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4690.325321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.034788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.205122] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4702.180141] LustreError: 158497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.226086] LustreError: 158497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4707.313066] LustreError: 158498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.662536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.174692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4713.057510] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4720.620581] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4725.028950] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4736.097036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4736.660768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.736804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4747.638936] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4754.502970] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4768.739761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4768.745783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.753867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4768.808495] Lustre: Skipped 1 previous similar message [ 4771.554472] Lustre: server umount lustre-OST0000 complete [ 4778.978990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.989777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.991076] Lustre: Skipped 1 previous similar message [ 4778.997543] Lustre: Skipped 2 previous similar messages [ 4782.639612] Lustre: server umount lustre-MDT0000 complete [ 4784.101072] LustreError: 158498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.117422] LustreError: 158498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4786.621884] LustreError: 158477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851345 with bad export cookie 4423229113788232400 [ 4786.624939] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.637646] LustreError: 158477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4787.114636] Lustre: server umount lustre-MDT0001 complete [ 4797.003552] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4800.256971] Key type lgssc unregistered [ 4800.641575] LNet: 161695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4800.667392] LNetError: 161695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4800.707581] LNet: Removed LNI 192.168.202.130@tcp [ 4801.881513] Key type .llcrypt unregistered [ 4801.884889] Key type ._llcrypt unregistered [ 4842.331956] Key type ._llcrypt registered [ 4842.334731] Key type .llcrypt registered [ 4842.446935] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4843.893811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4843.922644] alg: No test for adler32 (adler32-zlib) [ 4845.001873] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 4845.271182] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4847.047180] Key type lgssc registered [ 4848.087839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4856.357558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4856.394878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.001851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.333776] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4867.557404] LustreError: 163385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.579829] LustreError: 163385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4872.200608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.696121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4877.844302] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4884.175917] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4888.252838] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4896.193711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.777879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4902.915000] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4903.632635] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4910.900404] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4923.366610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4923.382269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.404067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4923.408547] Lustre: Skipped 1 previous similar message [ 4925.936586] Lustre: server umount lustre-OST0000 complete [ 4933.602246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.614422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.618140] Lustre: Skipped 2 previous similar messages [ 4933.652094] Lustre: Skipped 1 previous similar message [ 4936.705585] Lustre: server umount lustre-MDT0000 complete [ 4938.720082] LustreError: 164096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.741671] LustreError: 164096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4941.017317] LustreError: 163368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851499 with bad export cookie 348916442777677547 [ 4941.018189] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.026024] LustreError: 163368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4941.537504] Lustre: server umount lustre-MDT0001 complete [ 4954.171776] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4957.774915] Key type lgssc unregistered [ 4958.260810] LNet: 166566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4958.268071] LNetError: 166566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4958.292525] LNet: Removed LNI 192.168.202.130@tcp [ 4959.515061] Key type .llcrypt unregistered [ 4959.518893] Key type ._llcrypt unregistered [ 4986.417765] Key type ._llcrypt registered [ 4986.423794] Key type .llcrypt registered [ 4986.628551] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4988.185789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4988.276489] alg: No test for adler32 (adler32-zlib) [ 4989.376669] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 4989.657543] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4991.383191] Key type lgssc registered [ 4992.678606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5004.822893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5004.842790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.540521] Lustre: *** cfs_fail_loc=135, val=0*** [ 5006.680124] LustreError: 168077:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000 failed (-2) [ 5006.688228] LustreError: 168077:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.130@tcp: cfg command failed: rc = -2 [ 5006.705761] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5006.705761] [ 5006.718900] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5006.735292] LustreError: 168037:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5006.743923] LustreError: 168037:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 5006.755136] LustreError: 168037:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 5006.761427] LustreError: 168037:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 5006.833425] Lustre: server umount lustre-MDT0000 complete [ 5006.837243] LustreError: 168037:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5015.873303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.536117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.295510] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5026.788305] LustreError: 168448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.822028] LustreError: 168448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5031.909159] LustreError: 168449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.522385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.012980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5039.054482] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5048.631553] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5053.710638] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5057.861577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.878501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.885090] Lustre: Skipped 1 previous similar message [ 5063.136483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5063.138564] Lustre: Skipped 1 previous similar message [ 5068.263345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5068.266218] Lustre: Skipped 1 previous similar message [ 5070.303141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5070.460725] Lustre: server umount lustre-MDT0000 complete [ 5073.390180] LustreError: 168444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.418873] LustreError: 168444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5074.533239] LustreError: 168430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851633 with bad export cookie 4425212794314882102 [ 5074.543242] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.003078] Lustre: server umount lustre-MDT0001 complete [ 5087.859867] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5091.112083] Key type lgssc unregistered [ 5091.500645] LNet: 170529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5091.510801] LNetError: 170529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5091.529497] LNet: Removed LNI 192.168.202.130@tcp [ 5092.497393] Key type .llcrypt unregistered [ 5092.500693] Key type ._llcrypt unregistered [ 5117.243916] Key type ._llcrypt registered [ 5117.250205] Key type .llcrypt registered [ 5117.437923] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5117.767340] Key type .llcrypt unregistered [ 5117.769946] Key type ._llcrypt unregistered [ 5139.654424] Key type ._llcrypt registered [ 5139.656598] Key type .llcrypt registered [ 5139.761691] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5141.032785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5141.281628] alg: No test for adler32 (adler32-zlib) [ 5142.338224] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5142.551289] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5144.223170] Key type lgssc registered [ 5145.471601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5153.900715] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5153.931155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5169.504763] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5174.627173] LustreError: 172628:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 5201.311202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.495831] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5214.662573] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5222.649078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.342615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.424922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5228.515953] LustreError: 172653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.529908] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5228.543693] LustreError: 172653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5233.632937] LustreError: 172654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.717368] LustreError: 173554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.748448] LustreError: 173554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5239.506199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.967529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5245.807264] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5254.164392] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5258.936975] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5262.905416] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5274.594888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.613828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.649271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.623257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.629715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.663931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.934568] Lustre: server umount lustre-OST0000 complete [ 5285.857746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5285.881126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.901022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.911915] Lustre: Skipped 2 previous similar messages [ 5291.493039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5291.504258] Lustre: Skipped 3 previous similar messages [ 5291.893033] Lustre: server umount lustre-MDT0000 complete [ 5296.621159] LustreError: 173557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.651765] LustreError: 173557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5297.459897] LustreError: 174285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851856 with bad export cookie 12073820098222591790 [ 5297.460449] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5297.467109] LustreError: 174285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5297.974878] Lustre: server umount lustre-MDT0001 complete [ 5311.563832] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5314.516337] Key type lgssc unregistered [ 5314.801496] LNet: 176196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5314.807911] LNetError: 176196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5314.840762] LNet: Removed LNI 192.168.202.130@tcp [ 5315.794220] Key type .llcrypt unregistered [ 5315.797290] Key type ._llcrypt unregistered [ 5345.768235] Key type ._llcrypt registered [ 5345.771090] Key type .llcrypt registered [ 5345.869557] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5347.033225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5347.197995] alg: No test for adler32 (adler32-zlib) [ 5348.588870] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5349.104219] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5350.927219] Key type lgssc registered [ 5352.640732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5362.580394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5362.612216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.281514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.296819] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5373.415352] LustreError: 177594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.436126] LustreError: 177594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5378.535413] LustreError: 177595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.920611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.637119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5386.283714] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5392.972070] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5397.377231] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5405.026804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5405.487465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.624390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5411.403561] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5417.889251] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5423.836852] Lustre: Failing over lustre-MDT0000 [ 5424.100084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.103470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.127176] Lustre: Skipped 2 previous similar messages [ 5424.153675] Lustre: Skipped 2 previous similar messages [ 5425.112760] LustreError: 179271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5426.261262] Lustre: server umount lustre-MDT0000 complete [ 5429.218595] LustreError: 179751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.262783] LustreError: 179751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5434.338822] LustreError: 177590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.364475] LustreError: 177590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5444.319285] Lustre: 176905:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775851987/real 1775851987] req@000000009086a420 x1862115689261568/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775852003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5444.366298] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5444.408909] LustreError: 177590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.441826] LustreError: 177590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5448.141525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.189614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.275032] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5456.815499] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5460.051414] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5460.464915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5460.479363] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5460.531752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5465.066491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5473.537160] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5485.058511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5485.075619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.121479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5486.301984] Lustre: server umount lustre-OST0000 complete [ 5496.293058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.294419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5496.309348] Lustre: Skipped 1 previous similar message [ 5496.311085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5496.311091] Lustre: Skipped 2 previous similar messages [ 5498.809482] Lustre: server umount lustre-MDT0000 complete [ 5501.419836] LustreError: 177590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5501.456493] LustreError: 177590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5504.598115] LustreError: 177574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852063 with bad export cookie 11245740128952756831 [ 5504.609034] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5504.619307] LustreError: 177574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5505.277580] Lustre: server umount lustre-MDT0001 complete [ 5520.891419] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5524.165083] Key type lgssc unregistered [ 5524.594555] LNet: 182123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5524.608534] LNetError: 182123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5525.671316] LNet: Removed LNI 192.168.202.130@tcp [ 5526.995569] Key type .llcrypt unregistered [ 5527.001124] Key type ._llcrypt unregistered [ 5554.085912] Key type ._llcrypt registered [ 5554.089743] Key type .llcrypt registered [ 5554.213239] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5555.291571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5555.603839] alg: No test for adler32 (adler32-zlib) [ 5556.641495] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5556.920365] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5558.687279] Key type lgssc registered [ 5559.928447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5569.652649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5569.695549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.424877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.637504] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5581.807337] LustreError: 183522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.827995] LustreError: 183522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5586.585856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.026121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5592.469457] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5600.002787] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5605.172529] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5613.385563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5613.914853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5617.021496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5619.508775] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5625.416650] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5629.592493] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5633.136139] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5646.324304] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5657.061350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5657.067969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.070696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.098214] Lustre: Skipped 1 previous similar message [ 5661.297087] Lustre: server umount lustre-OST0000 complete [ 5667.297450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.319551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.322407] Lustre: Skipped 1 previous similar message [ 5672.429833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5672.455462] Lustre: Skipped 1 previous similar message [ 5672.746749] Lustre: server umount lustre-MDT0000 complete [ 5676.515267] LustreError: 183522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5676.531499] LustreError: 183522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5676.949870] LustreError: 183503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852235 with bad export cookie 11669546467438132880 [ 5676.951152] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5676.964290] LustreError: 183503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5677.900864] Lustre: server umount lustre-MDT0001 complete [ 5692.488817] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5695.947650] Key type lgssc unregistered [ 5696.232153] LNet: 186891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5696.243773] LNetError: 186891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5696.260044] LNet: Removed LNI 192.168.202.130@tcp [ 5697.181633] Key type .llcrypt unregistered [ 5697.190174] Key type ._llcrypt unregistered [ 5725.783762] Key type ._llcrypt registered [ 5725.785828] Key type .llcrypt registered [ 5725.876770] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5727.026247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5727.102330] alg: No test for adler32 (adler32-zlib) [ 5728.241601] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5728.497498] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5730.159199] Key type lgssc registered [ 5731.511594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5740.990574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5741.043338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.723658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.366412] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5753.319584] LustreError: 188287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.349493] LustreError: 188287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5756.469180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.807933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5762.368818] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5769.425364] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5773.886508] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5782.796499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5783.307996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5788.466644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5789.910468] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5797.278579] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5804.920170] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5813.845291] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5825.991146] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5840.636496] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5855.200263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.206562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.216863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.296266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.298081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.325563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.341753] Lustre: Skipped 1 previous similar message [ 5861.321228] Lustre: server umount lustre-OST0000 complete [ 5868.536274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.544049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5868.554547] Lustre: Skipped 1 previous similar message [ 5873.644749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.647352] Lustre: Skipped 1 previous similar message [ 5878.753524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.760392] Lustre: Skipped 1 previous similar message [ 5880.288518] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5880.510786] Lustre: server umount lustre-MDT0000 complete [ 5884.801563] LustreError: 188268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852443 with bad export cookie 5396376617350831663 [ 5884.803958] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.826483] LustreError: 188268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5885.394512] Lustre: server umount lustre-MDT0001 complete [ 5899.538286] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5903.578226] Key type lgssc unregistered [ 5903.965036] LNet: 193006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5903.970858] LNetError: 193006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5903.984437] LNet: Removed LNI 192.168.202.130@tcp [ 5904.916425] Key type .llcrypt unregistered [ 5904.918504] Key type ._llcrypt unregistered [ 5933.075635] Key type ._llcrypt registered [ 5933.077419] Key type .llcrypt registered [ 5933.204800] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5934.786273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5934.883832] alg: No test for adler32 (adler32-zlib) [ 5935.905611] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5936.160335] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5937.959890] Key type lgssc registered [ 5939.822352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5950.878507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5950.917633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.901804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.123304] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5963.266824] LustreError: 194402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.293412] LustreError: 194402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5968.362824] LustreError: 194403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.772528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.109354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5976.879975] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5984.028109] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5988.880610] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5996.086300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.491318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5997.573420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6002.415344] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6009.458722] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6026.219505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6026.232602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.241987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.251199] Lustre: Skipped 1 previous similar message [ 6026.270507] Lustre: Skipped 1 previous similar message [ 6031.336147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.353118] Lustre: Skipped 1 previous similar message [ 6031.891922] Lustre: server umount lustre-OST0000 complete [ 6036.451496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.464896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6036.477522] Lustre: Skipped 1 previous similar message [ 6036.500234] Lustre: Skipped 1 previous similar message [ 6041.572969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6041.578552] Lustre: Skipped 1 previous similar message [ 6046.688573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6046.694789] Lustre: Skipped 1 previous similar message [ 6050.271137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6050.452491] Lustre: server umount lustre-MDT0000 complete [ 6051.811239] LustreError: 194403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.828657] LustreError: 194403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6054.304468] LustreError: 194384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852613 with bad export cookie 2826092800257023160 [ 6054.315604] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6054.318334] LustreError: 194384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6054.751969] Lustre: server umount lustre-MDT0001 complete [ 6067.848804] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6070.512687] Key type lgssc unregistered [ 6070.774730] LNet: 197738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6070.783943] LNetError: 197738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6071.847559] LNet: Removed LNI 192.168.202.130@tcp [ 6072.636850] Key type .llcrypt unregistered [ 6072.639798] Key type ._llcrypt unregistered [ 6097.240878] Key type ._llcrypt registered [ 6097.244402] Key type .llcrypt registered [ 6097.492657] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6099.529513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.568417] alg: No test for adler32 (adler32-zlib) [ 6101.118472] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6101.591521] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6103.457937] Key type lgssc registered [ 6105.233905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6116.500719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6116.523323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.447533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.299327] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6128.611730] LustreError: 198989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.629784] LustreError: 198989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6133.309675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.880914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6138.860793] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6144.947865] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6149.978698] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6159.674732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6160.347115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.399879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6168.751627] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6175.464382] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6194.149623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6194.163178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.172793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.183118] Lustre: Skipped 1 previous similar message [ 6194.202341] Lustre: Skipped 1 previous similar message [ 6199.273075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6199.283540] Lustre: Skipped 1 previous similar message [ 6199.908643] Lustre: server umount lustre-OST0000 complete [ 6204.386304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.397281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.506271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.510993] Lustre: Skipped 2 previous similar messages [ 6210.422733] Lustre: server umount lustre-MDT0000 complete [ 6214.624694] LustreError: 198986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.650981] LustreError: 198986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6215.038520] LustreError: 198968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852773 with bad export cookie 7843728356985400265 [ 6215.041931] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6215.049340] LustreError: 198968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6215.664963] Lustre: server umount lustre-MDT0001 complete [ 6227.032583] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6229.691913] Key type lgssc unregistered [ 6230.019755] LNet: 202321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6230.043894] LNetError: 202321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6231.081249] LNet: Removed LNI 192.168.202.130@tcp [ 6231.835282] Key type .llcrypt unregistered [ 6231.837338] Key type ._llcrypt unregistered [ 6255.592761] Key type ._llcrypt registered [ 6255.598342] Key type .llcrypt registered [ 6255.777343] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6256.988549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6257.015780] alg: No test for adler32 (adler32-zlib) [ 6258.222674] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6258.510435] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6260.327906] Key type lgssc registered [ 6261.599639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6270.311979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6270.335721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6271.922399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6277.383735] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6282.215204] LustreError: 203569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.233827] LustreError: 203569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6287.331354] LustreError: 203570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.814155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.455639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.734439] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6300.006726] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6305.725555] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6315.250661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6315.726887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6316.796781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6324.208394] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6331.982141] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6341.152846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6341.247616] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6341.454434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.917714] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6354.154298] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6356.258157] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6368.831770] Lustre: Permanently deactivating lustre-OST0001 [ 6368.859286] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6379.145152] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6391.496624] Lustre: Permanently reactivating lustre-OST0001 [ 6391.512312] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6391.516736] Lustre: Skipped 2 previous similar messages [ 6399.192758] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.215678] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6399.234965] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6399.275664] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6399.358858] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6409.696935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.705960] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.714525] LustreError: Skipped 1 previous similar message [ 6409.725042] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6409.758746] Lustre: Skipped 2 previous similar messages [ 6411.258532] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6411.271081] Lustre: Skipped 2 previous similar messages [ 6413.743785] Lustre: server umount lustre-OST0001 complete [ 6416.384475] LustreError: 205731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.413560] LustreError: 205731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6419.424943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.436079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.449729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6421.482448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6421.487470] LustreError: 205165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.499162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6421.499188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.530690] LustreError: 205165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6426.598892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.601874] LustreError: 205167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.608589] Lustre: Skipped 2 previous similar messages [ 6426.620189] LustreError: 205167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6433.248070] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6433.588602] Lustre: server umount lustre-OST0000 complete [ 6441.952706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.965496] Lustre: Skipped 1 previous similar message [ 6441.966882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.994876] Lustre: Skipped 2 previous similar messages [ 6452.191217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6452.430907] Lustre: server umount lustre-MDT0000 complete [ 6456.312694] LustreError: 203549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853015 with bad export cookie 3262719958367771643 [ 6456.315387] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.332602] LustreError: 203549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6456.928271] Lustre: server umount lustre-MDT0001 complete [ 6469.216538] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6471.903710] Key type lgssc unregistered [ 6472.276725] LNet: 208225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6472.280445] LNetError: 208225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6472.298877] LNet: Removed LNI 192.168.202.130@tcp [ 6473.150201] Key type .llcrypt unregistered [ 6473.152161] Key type ._llcrypt unregistered [ 6516.527896] Key type ._llcrypt registered [ 6516.530033] Key type .llcrypt registered [ 6516.660360] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6518.551403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6518.636235] alg: No test for adler32 (adler32-zlib) [ 6519.864416] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6520.088217] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6521.720768] Key type lgssc registered [ 6523.342819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6531.956643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6531.982047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.262987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6533.285374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6533.475049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.334952] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6548.738956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.831421] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6548.890865] Lustre: 210416:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6549.285953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6554.492969] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6562.437681] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6567.513328] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6575.133906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6575.401502] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6575.583045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.873565] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6584.947499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6588.801623] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6592.045692] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6600.488679] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6606.764560] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6614.197258] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6624.268581] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6646.705373] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6646.720325] Lustre: Skipped 2 previous similar messages [ 6680.844370] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6680.849237] Lustre: Skipped 3 previous similar messages [ 6754.916807] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6754.923953] Lustre: Skipped 9 previous similar messages [ 6764.000593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.007430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6764.008963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6764.036843] Lustre: Skipped 1 previous similar message [ 6769.125404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6769.135349] Lustre: Skipped 2 previous similar messages [ 6769.823959] Lustre: server umount lustre-OST0000 complete [ 6779.361722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6779.370036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6779.375423] Lustre: Skipped 1 previous similar message [ 6779.398204] Lustre: Skipped 1 previous similar message [ 6781.809781] Lustre: server umount lustre-MDT0000 complete [ 6785.840394] LustreError: 213682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853344 with bad export cookie 14437226049149957724 [ 6785.843186] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.848702] LustreError: 213682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6786.207460] Lustre: server umount lustre-MDT0001 complete [ 6799.577025] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6802.847768] Key type lgssc unregistered [ 6803.173405] LNet: 214855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6803.183307] LNetError: 214855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6803.198308] LNet: Removed LNI 192.168.202.130@tcp [ 6803.894352] Key type .llcrypt unregistered [ 6803.896526] Key type ._llcrypt unregistered [ 6833.590657] Key type ._llcrypt registered [ 6833.592769] Key type .llcrypt registered [ 6833.688686] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6834.766184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6834.854280] alg: No test for adler32 (adler32-zlib) [ 6835.974918] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6836.205270] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6837.879354] Key type lgssc registered [ 6839.872368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6848.721442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6848.742950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.472279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.201427] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6860.771848] LustreError: 216253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.806693] LustreError: 216253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6865.526758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.124739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6870.748825] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6877.817342] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6882.574596] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6893.523396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6894.092805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6896.118484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6902.073091] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6910.403148] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6936.034695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.035940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.057483] Lustre: Skipped 1 previous similar message [ 6938.893431] Lustre: server umount lustre-OST0000 complete [ 6946.272492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.296872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.313094] Lustre: Skipped 2 previous similar messages [ 6950.295377] Lustre: server umount lustre-MDT0000 complete [ 6951.409931] LustreError: 216250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.449472] LustreError: 216250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6955.520174] LustreError: 218499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853514 with bad export cookie 133498548896795947 [ 6955.533539] LustreError: 218499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6955.535204] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6956.173464] Lustre: server umount lustre-MDT0001 complete [ 6969.733299] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6972.762653] Key type lgssc unregistered [ 6973.056905] LNet: 219592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6973.061312] LNetError: 219592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6974.124220] LNet: Removed LNI 192.168.202.130@tcp [ 6975.080428] Key type .llcrypt unregistered [ 6975.084911] Key type ._llcrypt unregistered [ 7005.419829] Key type ._llcrypt registered [ 7005.422371] Key type .llcrypt registered [ 7005.529873] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7006.720765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7006.808131] alg: No test for adler32 (adler32-zlib) [ 7007.987539] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 7008.296985] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7009.972458] Key type lgssc registered [ 7011.343391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7283.791634] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7287.089878] Key type lgssc unregistered [ 7287.432788] LNet: 221594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7287.445645] LNetError: 221594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7287.472207] LNet: Removed LNI 192.168.202.130@tcp [ 7288.405083] Key type .llcrypt unregistered [ 7288.412043] Key type ._llcrypt unregistered [ 7318.418149] Key type ._llcrypt registered [ 7318.429713] Key type .llcrypt registered [ 7318.561260] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 7340.587367] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7342.189367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7342.220571] alg: No test for adler32 (adler32-zlib) [ 7343.245411] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 7343.438944] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7345.120897] Key type lgssc registered [ 7346.889279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7386.151578] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7147 sec ========== 16:45:43 (1775853943) [ 7388.391984] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:45:45 (1775853945) === [ 7392.193713] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:45:49 (1775853949) === [ 7440.896621] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7444.189416] Key type lgssc unregistered [ 7444.474161] LNet: 226785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7444.487491] LNetError: 226785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7444.498586] LNet: Removed LNI 192.168.202.130@tcp [ 7445.449722] Key type .llcrypt unregistered [ 7445.453300] Key type ._llcrypt unregistered