[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492685764 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002349] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006501] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011033] SELinux: Initializing. [ 0.012056] *** VALIDATE selinux *** [ 0.021078] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025853] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] *** VALIDATE tmpfs *** [ 0.031485] *** VALIDATE proc *** [ 0.033257] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036194] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043083] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.045976] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054236] rcu: Hierarchical SRCU implementation. [ 0.056542] smp: Bringing up secondary CPUs ... [ 0.057606] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.061318] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143174] node 0 deferred pages initialised in 74ms [ 0.145152] devtmpfs: initialized [ 0.147260] x86/mm: Memory block size: 128MB [ 0.150896] gcov: version magic: 0x41383552 [ 0.152156] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158466] pinctrl core: initialized pinctrl subsystem [ 0.160218] [ 0.161010] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166013] ** ** [ 0.168011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170015] ** ** [ 0.172014] ** This means that this kernel is built to expose internal ** [ 0.174015] ** IOMMU data structures, which may compromise security on ** [ 0.177014] ** your system. ** [ 0.179012] ** ** [ 0.182017] ** If you see this message and you are not debugging the ** [ 0.184018] ** kernel, report this immediately to your vendor! ** [ 0.186017] ** ** [ 0.189012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191014] ************************************************************* [ 0.193409] NET: Registered protocol family 16 [ 0.195552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203028] cpuidle: using governor menu [ 0.205000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206355] PCI: Using configuration type 1 for base access [ 0.208097] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218141] cryptd: max_cpu_qlen set to 1000 [ 0.221215] ACPI: Added _OSI(Module Device) [ 0.223028] ACPI: Added _OSI(Processor Device) [ 0.225015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229398] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235293] ACPI: Interpreter enabled [ 0.237052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238009] ACPI: Using IOAPIC for interrupt routing [ 0.239137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249501] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258986] acpiphp: Slot [2] registered [ 0.260062] acpiphp: Slot [5] registered [ 0.261018] acpiphp: Slot [6] registered [ 0.262068] acpiphp: Slot [7] registered [ 0.263041] acpiphp: Slot [8] registered [ 0.263980] acpiphp: Slot [9] registered [ 0.265091] acpiphp: Slot [10] registered [ 0.266020] acpiphp: Slot [3] registered [ 0.266991] acpiphp: Slot [4] registered [ 0.268062] acpiphp: Slot [11] registered [ 0.268987] acpiphp: Slot [12] registered [ 0.269058] acpiphp: Slot [13] registered [ 0.270060] acpiphp: Slot [14] registered [ 0.270962] acpiphp: Slot [15] registered [ 0.272060] acpiphp: Slot [16] registered [ 0.272885] acpiphp: Slot [17] registered [ 0.274050] acpiphp: Slot [18] registered [ 0.274932] acpiphp: Slot [19] registered [ 0.275053] acpiphp: Slot [20] registered [ 0.276031] acpiphp: Slot [21] registered [ 0.276897] acpiphp: Slot [22] registered [ 0.278049] acpiphp: Slot [23] registered [ 0.278935] acpiphp: Slot [24] registered [ 0.280083] acpiphp: Slot [25] registered [ 0.281077] acpiphp: Slot [26] registered [ 0.282110] acpiphp: Slot [27] registered [ 0.283072] acpiphp: Slot [28] registered [ 0.283969] acpiphp: Slot [29] registered [ 0.285075] acpiphp: Slot [30] registered [ 0.285921] acpiphp: Slot [31] registered [ 0.287049] PCI host bridge to bus 0000:00 [ 0.287900] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.290012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298209] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305287] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.322063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.324020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.327019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.329028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.332439] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334583] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.336036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338590] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.351012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.356787] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.360093] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.375016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.416419] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453049] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.461650] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.468038] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.475017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.501685] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.509015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.514013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.528017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.547196] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.557024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.568018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.599693] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.609016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.617020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.635015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.646009] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648346] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.649309] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.652392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.661038] iommu: Default domain type: Passthrough [ 0.663504] SCSI subsystem initialized [ 0.665126] ACPI: bus type USB registered [ 0.666128] usbcore: registered new interface driver usbfs [ 0.669074] usbcore: registered new interface driver hub [ 0.671079] usbcore: registered new device driver usb [ 0.673211] pps_core: LinuxPPS API ver. 1 registered [ 0.675014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.678052] PTP clock support registered [ 0.680302] EDAC MC: Ver: 3.0.0 [ 0.681502] PCI: Using ACPI for IRQ routing [ 0.682847] NetLabel: Initializing [ 0.683008] NetLabel: domain hash size = 128 [ 0.684010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.685169] NetLabel: unlabeled traffic allowed by default [ 0.686277] vgaarb: loaded [ 0.687333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.696191] clocksource: Switched to clocksource kvm-clock [ 0.807469] VFS: Disk quotas dquot_6.6.0 [ 0.809153] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.811864] *** VALIDATE ramfs *** [ 0.813185] *** VALIDATE hugetlbfs *** [ 0.815265] pnp: PnP ACPI init [ 0.817700] pnp: PnP ACPI: found 6 devices [ 0.833668] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.837339] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840416] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.843557] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.846546] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.849371] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.852689] NET: Registered protocol family 2 [ 0.855615] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.860689] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.864558] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.869980] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.873195] TCP: Hash tables configured (established 65536 bind 65536) [ 0.876385] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878580] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880659] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883826] NET: Registered protocol family 1 [ 0.886083] RPC: Registered named UNIX socket transport module. [ 0.888417] RPC: Registered udp transport module. [ 0.890323] RPC: Registered tcp transport module. [ 0.892282] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894923] NET: Registered protocol family 44 [ 0.896968] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.899419] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.902190] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.904886] PCI: CLS 0 bytes, default 64 [ 0.906580] Unpacking initramfs... [ 2.352116] debug: unmapping init [mem 0xffff9c3b7cc54000-0xffff9c3b7ffbffff] [ 2.355580] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.357678] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.359749] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.862439] Initialise system trusted keyrings [ 2.863632] Key type blacklist registered [ 2.866319] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.875225] zbud: loaded [ 2.878232] *** VALIDATE nfs *** [ 2.878920] *** VALIDATE nfs4 *** [ 2.880515] pstore: using deflate compression [ 2.884260] Platform Keyring initialized [ 2.978280] NET: Registered protocol family 38 [ 2.980129] Key type asymmetric registered [ 2.982841] Asymmetric key parser 'x509' registered [ 2.985535] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.989296] io scheduler mq-deadline registered [ 2.991327] io scheduler kyber registered [ 2.993997] io scheduler bfq registered [ 2.995963] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.998681] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.001419] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.004635] ACPI: Power Button [PWRF] [ 3.010894] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.018053] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.031736] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.039472] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.065136] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091199] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.121698] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.126347] Non-volatile memory driver v1.3 [ 3.127437] Linux agpgart interface v0.103 [ 3.155813] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.157879] vda: detected capacity change from 0 to 75071488 [ 3.168748] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.170559] vdb: detected capacity change from 0 to 1073741824 [ 3.181930] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.183900] vdc: detected capacity change from 0 to 2621440000 [ 3.194764] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.196506] vdd: detected capacity change from 0 to 2621440000 [ 3.208383] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.210162] vde: detected capacity change from 0 to 4294967296 [ 3.223991] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.225684] vdf: detected capacity change from 0 to 4294967296 [ 3.230305] libphy: Fixed MDIO Bus: probed [ 3.235671] usbcore: registered new interface driver usbserial_generic [ 3.238548] usbserial: USB Serial support registered for generic [ 3.240321] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.243576] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.245323] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.247488] mousedev: PS/2 mouse device common for all mice [ 3.252189] rtc_cmos 00:05: RTC can wake from S4 [ 3.257282] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.257314] rtc_cmos 00:05: registered as rtc0 [ 3.261307] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.263879] intel_pstate: CPU model not supported [ 3.266328] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.267956] hid: raw HID events driver (C) Jiri Kosina [ 3.270807] usbcore: registered new interface driver usbhid [ 3.271912] usbhid: USB HID core driver [ 3.272064] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.272717] drop_monitor: Initializing network drop monitor service [ 3.276034] Initializing XFRM netlink socket [ 3.277050] NET: Registered protocol family 10 [ 3.278541] Segment Routing with IPv6 [ 3.279694] NET: Registered protocol family 17 [ 3.281140] mpls_gso: MPLS GSO support [ 3.285649] RAS: Correctable Errors collector initialized. [ 3.287130] AVX version of gcm_enc/dec engaged. [ 3.288319] AES CTR mode by8 optimization enabled [ 3.340977] sched_clock: Marking stable (3340955313, 0)->(4231720528, -890765215) [ 3.343933] registered taskstats version 1 [ 3.345678] Loading compiled-in X.509 certificates [ 3.347290] zswap: loaded using pool lzo/zbud [ 3.364602] Key type big_key registered [ 3.374417] Key type encrypted registered [ 3.375621] ima: No TPM chip found, activating TPM-bypass! [ 3.377282] ima: Allocated hash algorithm: sha1 [ 3.378480] ima: No architecture policies found [ 3.379566] evm: Initialising EVM extended attributes: [ 3.380905] evm: security.selinux [ 3.381777] evm: security.ima [ 3.382535] evm: security.capability [ 3.383389] evm: HMAC attrs: 0x1 [ 3.384883] rtc_cmos 00:05: setting system clock to 2026-08-26 22:18:03 UTC (1787782683) [ 3.388786] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 3.391125] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 3.402059] Write protecting the kernel read-only data: 28672k [ 3.404114] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 3.405968] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 3.434331] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.441556] systemd[1]: Detected virtualization kvm. [ 3.443523] systemd[1]: Detected architecture x86-64. [ 3.445310] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.471900] systemd[1]: No hostname configured. [ 3.473754] systemd[1]: Set hostname to . [ 3.475898] random: systemd: uninitialized urandom read (16 bytes read) [ 3.478584] systemd[1]: Initializing machine ID from random generator. [ 3.523829] random: ln: uninitialized urandom read (6 bytes read) [ 3.595202] random: systemd: uninitialized urandom read (16 bytes read) [ 3.597186] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.600623] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.605365] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.119527] device-mapper: uevent: version 1.0.3 [ 4.121280] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.693319] random: fast init done [ 4.706706] virtio_net virtio0 ens2: renamed from eth0 [ 4.720355] scsi host0: ata_piix [ 4.735556] scsi host1: ata_piix [ 4.737131] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.739099] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.891172] random: crng init done [ 8.892805] random: 7 urandom warning(s) missed due to ratelimiting [ 8.953249] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.517482] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.670804] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.942323] SELinux: Disabled at runtime. [ 11.003850] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.011970] systemd[1]: Detected virtualization kvm. [ 11.013674] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.565148] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.568725] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.574704] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.578920] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.583290] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.591782] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.597141] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 11.636960] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.109186] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.472244] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.485315] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.679586] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.688537] EDAC sbridge: Ver: 1.1.2 [ 14.265825] Key type dns_resolver registered [ 14.556738] NFS: Registering the id_resolver key type [ 14.558943] Key type id_resolver registered [ 14.560610] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg234-server login: [ 39.763851] libcfs: loading out-of-tree module taints kernel. [ 39.782622] Key type ._llcrypt registered [ 39.783739] Key type .llcrypt registered [ 39.834201] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 51.293763] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 53.048412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.058174] alg: No test for adler32 (adler32-zlib) [ 54.515709] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 55.470572] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 57.336165] Key type lgssc registered [ 59.956697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.078221] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.006042] hrtimer: interrupt took 2631623 ns [ 116.684948] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 131.382670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.452062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.814841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.860110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.960049] Lustre: lustre-MDT0000: new disk, initializing [ 133.090226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.125935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.908277] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.680812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.800274] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 151.829722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 151.833798] Lustre: Skipped 1 previous similar message [ 151.940445] Lustre: lustre-MDT0001: new disk, initializing [ 152.061837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.106533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 152.115579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 156.218830] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.825799] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 170.939571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.318046] Lustre: lustre-OST0000: new disk, initializing [ 171.326816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.333951] Lustre: 8418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.426370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.804909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.817052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.980611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.892234] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.891853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.079706] Lustre: lustre-OST0001: new disk, initializing [ 197.083269] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.094116] Lustre: 9489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.190262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.506973] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.430294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.440748] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.522574] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 216.746620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.922529] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.341750] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.854957] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing yml_node [ 239.862226] Lustre: DEBUG MARKER: Client: 2.17.57.55 [ 242.490631] Lustre: DEBUG MARKER: MDS: 2.17.57.55 [ 245.630518] Lustre: DEBUG MARKER: OSS: 2.17.57.55 [ 247.320588] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:22:05 EDT 2026 [ 265.234230] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 266.828783] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 284.641146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.644906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.667546] Lustre: Skipped 3 previous similar messages [ 284.681883] Lustre: Skipped 2 previous similar messages [ 289.380901] Lustre: server umount lustre-MDT0000 complete [ 293.080223] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782973 with bad export cookie 16769998866010102781 [ 293.082551] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.086270] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 294.881644] LustreError: 6487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.883928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.884405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.884409] Lustre: Skipped 1 previous similar message [ 294.895288] LustreError: 6487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 294.902149] Lustre: Skipped 1 previous similar message [ 298.934309] LustreError: 9497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.936048] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.958324] LustreError: 9497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 298.994876] Lustre: Skipped 2 previous similar messages [ 299.546558] Lustre: server umount lustre-MDT0001 complete [ 304.072968] Lustre: server umount lustre-OST0000 complete [ 308.248553] Lustre: server umount lustre-OST0001 complete [ 314.655375] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 325.458361] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 362.391845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.549719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.579496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.668988] Lustre: lustre-MDT0000: new disk, initializing [ 362.733832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.751277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.811950] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 379.396433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.620323] Lustre: 15043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.692160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 379.698408] Lustre: Skipped 1 previous similar message [ 379.842409] Lustre: lustre-MDT0001: new disk, initializing [ 379.986151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.043903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.063850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 385.239987] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 395.443977] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 401.583624] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 408.687439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.089748] Lustre: lustre-OST0000: new disk, initializing [ 409.110059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 409.120967] Lustre: 17011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 409.240807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.030679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 411.042860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 411.207755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.894395] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 426.022153] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 431.098180] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 431.377915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 436.602695] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 436.786156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.834888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.853807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.857603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.900575] Lustre: Skipped 1 previous similar message [ 445.381659] Lustre: server umount lustre-OST0000 complete [ 451.556206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 451.562170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.583282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.585835] Lustre: Skipped 1 previous similar message [ 455.847792] Lustre: server umount lustre-MDT0000 complete [ 457.185573] LustreError: 15051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.215607] LustreError: 15051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 459.800756] LustreError: 15035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783139 with bad export cookie 16769998866010104314 [ 459.807758] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 459.812808] LustreError: 15035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 460.495968] Lustre: server umount lustre-MDT0001 complete [ 464.614693] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 18:25:43 (1787783143) [ 470.873041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.444588] LustreError: 19276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.531551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.085985] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 476.653076] LustreError: 19277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.647947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.088326] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 498.295296] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 504.853784] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.364408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.667964] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 528.263818] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 540.137850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 540.143830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.156417] Lustre: Skipped 1 previous similar message [ 540.161148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.167052] Lustre: Skipped 2 previous similar messages [ 545.449960] Lustre: server umount lustre-OST0000 complete [ 551.908375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.940619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.950348] Lustre: Skipped 2 previous similar messages [ 551.976413] LustreError: Skipped 1 previous similar message [ 555.859542] Lustre: server umount lustre-MDT0000 complete [ 557.026181] LustreError: 21318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.056391] LustreError: 21318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 560.721657] LustreError: 19258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783240 with bad export cookie 16769998866010105189 [ 560.733263] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.740580] LustreError: 19258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 561.279780] Lustre: server umount lustre-MDT0001 complete [ 575.397481] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 578.948526] Key type lgssc unregistered [ 579.131669] LNet: 22815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.152589] LNetError: 22815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.181400] LNet: Removed LNI 192.168.202.134@tcp [ 580.365728] Key type .llcrypt unregistered [ 580.372872] Key type ._llcrypt unregistered [ 607.825727] Key type ._llcrypt registered [ 607.829231] Key type .llcrypt registered [ 607.935858] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 609.230899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.301673] alg: No test for adler32 (adler32-zlib) [ 610.453430] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 610.761842] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 612.473754] Key type lgssc registered [ 613.823114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 622.533772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 622.549038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.969092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.885995] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 634.348521] LustreError: 24231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.359421] LustreError: 24231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 637.986796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.646773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 643.495185] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 652.356869] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 659.085883] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 666.766412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.252559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.312155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 674.209515] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 680.648658] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 699.371379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.375328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.384320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 704.494050] LustreError: 26053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.668299] Lustre: server umount lustre-OST0000 complete [ 709.630984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.641050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.651985] Lustre: Skipped 2 previous similar messages [ 709.659932] Lustre: Skipped 1 previous similar message [ 714.724180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.728553] Lustre: Skipped 1 previous similar message [ 714.981119] Lustre: server umount lustre-MDT0000 complete [ 718.568731] LustreError: 26075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783398 with bad export cookie 9927842483506242170 [ 718.591324] LustreError: 26075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 718.597113] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.087482] Lustre: server umount lustre-MDT0001 complete [ 731.391159] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 733.937665] Key type lgssc unregistered [ 734.177785] LNet: 28210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.183853] LNetError: 28210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.197705] LNet: Removed LNI 192.168.202.134@tcp [ 735.116273] Key type .llcrypt unregistered [ 735.118283] Key type ._llcrypt unregistered [ 759.390325] Key type ._llcrypt registered [ 759.400990] Key type .llcrypt registered [ 759.536350] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 761.046965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 761.110827] alg: No test for adler32 (adler32-zlib) [ 762.626818] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 763.030917] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 764.898828] Key type lgssc registered [ 767.276949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.239858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 777.272967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.743900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.854288] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 787.948618] LustreError: 29630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.962036] LustreError: 29630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 792.679712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.088767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 798.510968] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 805.670163] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 812.007271] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 826.123609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 826.465856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.744619] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 840.376328] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 848.004166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 863.724253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.726609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 863.750878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.761212] Lustre: Skipped 1 previous similar message [ 866.048939] Lustre: server umount lustre-OST0000 complete [ 873.954240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.962534] Lustre: Skipped 1 previous similar message [ 873.966041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.972104] Lustre: Skipped 1 previous similar message [ 876.363792] Lustre: server umount lustre-MDT0000 complete [ 879.073731] LustreError: 32137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.088703] LustreError: 32137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 879.715553] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783559 with bad export cookie 17366748850258730821 [ 879.724816] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.183901] Lustre: server umount lustre-MDT0001 complete [ 892.607197] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 895.456857] Key type lgssc unregistered [ 895.745757] LNet: 33531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.753836] LNetError: 33531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.775788] LNet: Removed LNI 192.168.202.134@tcp [ 896.659364] Key type .llcrypt unregistered [ 896.660936] Key type ._llcrypt unregistered [ 919.501718] Key type ._llcrypt registered [ 919.503953] Key type .llcrypt registered [ 919.596437] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 920.671306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 920.797260] alg: No test for adler32 (adler32-zlib) [ 921.901627] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 922.344618] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 924.056170] Key type lgssc registered [ 925.218412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 932.980339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 933.012920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.443453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.772203] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 944.619065] LustreError: 34947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.653478] LustreError: 34947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 946.672879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.050140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 951.934781] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 959.546609] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 965.343632] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 972.555489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 972.710945] LustreError: 36768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.067644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.973324] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 979.506276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 986.612790] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 998.881755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.898506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.925408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.916704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.919723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.928400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.036069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1005.042071] Lustre: Skipped 2 previous similar messages [ 1010.147156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.150119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 1010.169448] Lustre: Skipped 1 previous similar message [ 1010.510424] Lustre: server umount lustre-OST0000 complete [ 1015.270658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.274530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.281240] Lustre: Skipped 1 previous similar message [ 1015.305609] Lustre: Skipped 1 previous similar message [ 1020.388682] LustreError: 35949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.455977] Lustre: server umount lustre-MDT0000 complete [ 1023.843247] LustreError: 34927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783703 with bad export cookie 7560068942426983153 [ 1023.847647] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.851375] LustreError: 34927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1024.168278] Lustre: server umount lustre-MDT0001 complete [ 1034.077782] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1036.675816] Key type lgssc unregistered [ 1036.961174] LNet: 38484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.966890] LNetError: 38484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.996262] LNet: Removed LNI 192.168.202.134@tcp [ 1037.858989] Key type .llcrypt unregistered [ 1037.861721] Key type ._llcrypt unregistered [ 1069.328168] Key type ._llcrypt registered [ 1069.332941] Key type .llcrypt registered [ 1069.477289] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1071.322983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.346396] alg: No test for adler32 (adler32-zlib) [ 1072.551211] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1072.972242] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1074.785812] Key type lgssc registered [ 1076.562811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1086.942504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1086.972063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.592344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.621613] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1098.740594] LustreError: 39901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.771439] LustreError: 39901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1102.161482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.739980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.075279] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1116.800882] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1123.794744] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1131.395814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1131.789714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.885845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1139.307515] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1147.051935] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1153.655730] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1154.530103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.535933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.545612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.606410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.610313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.623279] Lustre: Skipped 1 previous similar message [ 1158.784919] Lustre: server umount lustre-OST0000 complete [ 1179.493692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.534759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.545121] Lustre: Skipped 1 previous similar message [ 1186.068953] Lustre: server umount lustre-MDT0000 complete [ 1187.820692] LustreError: 39896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.855920] LustreError: 39896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1190.439855] LustreError: 39880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783870 with bad export cookie 5166830170746979303 [ 1190.458648] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.460690] LustreError: 39880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1196.900277] Lustre: server umount lustre-MDT0001 complete [ 1211.312224] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1214.746124] Key type lgssc unregistered [ 1215.116918] LNet: 43613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.133203] LNetError: 43613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.173043] LNet: Removed LNI 192.168.202.134@tcp [ 1216.236516] Key type .llcrypt unregistered [ 1216.238719] Key type ._llcrypt unregistered [ 1242.069635] Key type ._llcrypt registered [ 1242.071623] Key type .llcrypt registered [ 1242.250476] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1243.522123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1243.738941] alg: No test for adler32 (adler32-zlib) [ 1245.050764] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1245.467464] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1247.168178] Key type lgssc registered [ 1248.623736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1256.360858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1256.383212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.841985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.008860] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1267.177703] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.193069] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1270.789715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.187402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.495198] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1283.259517] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1288.340552] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1295.125837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1295.634918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.456277] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1302.845365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1310.525326] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1318.089863] Lustre: server umount lustre-MDT0000 complete [ 1318.391236] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.394807] LustreError: 46711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.415154] Lustre: Skipped 1 previous similar message [ 1321.691851] LustreError: 45010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784001 with bad export cookie 1262679364146397947 [ 1321.701219] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.709618] LustreError: 45010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1323.490661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.493256] LustreError: 46712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.505204] Lustre: Skipped 1 previous similar message [ 1323.508642] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1323.546610] LustreError: 46712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1326.497837] LustreError: 46712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.498227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1326.515153] LustreError: 46712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1326.520460] Lustre: Skipped 1 previous similar message [ 1328.113848] Lustre: server umount lustre-MDT0001 complete [ 1349.521901] Lustre: server umount lustre-OST0000 complete [ 1366.890887] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1370.622142] Key type lgssc unregistered [ 1371.031101] LNet: 48889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.041792] LNetError: 48889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.059049] LNet: Removed LNI 192.168.202.134@tcp [ 1372.159424] Key type .llcrypt unregistered [ 1372.162242] Key type ._llcrypt unregistered [ 1398.789838] Key type ._llcrypt registered [ 1398.793733] Key type .llcrypt registered [ 1398.987180] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1400.586874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.764493] alg: No test for adler32 (adler32-zlib) [ 1401.937179] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1402.203482] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1403.912169] Key type lgssc registered [ 1405.398267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1414.545713] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1414.588361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.178333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1435.364584] LustreError: 50279:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1462.176821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.183751] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1480.883589] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1648.659661] Lustre: server umount lustre-OST0000 complete [ 1667.389365] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1671.034251] Key type lgssc unregistered [ 1671.557878] LNet: 51958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.570603] LNetError: 51958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.620908] LNet: Removed LNI 192.168.202.134@tcp [ 1673.757549] Key type .llcrypt unregistered [ 1673.784144] Key type ._llcrypt unregistered [ 1701.094309] Key type ._llcrypt registered [ 1701.096848] Key type .llcrypt registered [ 1701.217187] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1702.704918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1702.801808] alg: No test for adler32 (adler32-zlib) [ 1703.868475] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1704.140593] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1705.864208] Key type lgssc registered [ 1707.200704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.412343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1715.436088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.991666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.711680] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1727.468702] LustreError: 53370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.496760] LustreError: 53370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1732.602522] LustreError: 53371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.862471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.670180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.012799] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1748.600817] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1755.555266] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1763.381669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1764.129362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1770.789978] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1779.816682] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1783.785448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.809702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.813951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.710664] Lustre: server umount lustre-OST0000 complete [ 1789.928530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1789.967119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.002120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.969522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.971721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.018579] Lustre: Skipped 1 previous similar message [ 1795.899530] Lustre: server umount lustre-MDT0000 complete [ 1797.091558] LustreError: 53365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.120521] LustreError: 53365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1799.887560] LustreError: 53353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784480 with bad export cookie 13867957435958659169 [ 1799.908957] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.914897] LustreError: 53353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1800.493262] Lustre: server umount lustre-MDT0001 complete [ 1812.102084] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1814.690996] Key type lgssc unregistered [ 1814.998436] LNet: 56904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.002913] LNetError: 56904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.016575] LNet: Removed LNI 192.168.202.134@tcp [ 1815.929522] Key type .llcrypt unregistered [ 1815.931449] Key type ._llcrypt unregistered [ 1841.923339] Key type ._llcrypt registered [ 1841.928134] Key type .llcrypt registered [ 1842.126472] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1843.468232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1843.722975] alg: No test for adler32 (adler32-zlib) [ 1844.779111] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1845.062834] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1846.720229] Key type lgssc registered [ 1848.195923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1858.090890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1858.109047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.696664] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1878.817693] LustreError: 58292:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1905.504286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.116757] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1919.204428] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1926.090030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.255612] LustreError: 57628:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008120b8f0 x1874627946088704/t0(0) o250->MGC192.168.202.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1926.804900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.898042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1931.513889] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1932.266426] LustreError: 58317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.294235] LustreError: 58317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1937.384380] LustreError: 58318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.478503] LustreError: 59328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.508941] LustreError: 59328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1940.918962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.423717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.128875] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1954.385364] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.270317] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1962.977513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.986787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.998414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1967.075943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1967.095720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.097349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.378697] Lustre: server umount lustre-OST0000 complete [ 1992.676677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1992.677635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.698777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.709539] Lustre: Skipped 1 previous similar message [ 1992.733670] Lustre: Skipped 2 previous similar messages [ 1997.793515] LustreError: 59333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.814232] LustreError: 59333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1997.832223] Lustre: server umount lustre-MDT0000 complete [ 2002.519283] LustreError: 58300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784682 with bad export cookie 18315649245590071865 [ 2002.523532] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.533687] LustreError: 58300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2008.968679] Lustre: server umount lustre-MDT0001 complete [ 2020.014721] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2023.234939] Key type lgssc unregistered [ 2023.525574] LNet: 62039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.549328] LNetError: 62039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.567863] LNet: Removed LNI 192.168.202.134@tcp [ 2024.582886] Key type .llcrypt unregistered [ 2024.585274] Key type ._llcrypt unregistered [ 2047.544753] Key type ._llcrypt registered [ 2047.546790] Key type .llcrypt registered [ 2047.693613] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2048.968448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2048.994042] alg: No test for adler32 (adler32-zlib) [ 2050.094163] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2050.335105] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2052.008222] Key type lgssc registered [ 2053.301805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2062.170590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2062.196174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.931114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.385925] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2074.088945] LustreError: 63463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.115769] LustreError: 63463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2078.501445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.937884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.181602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2093.617894] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.994879] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2106.464985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2106.899192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.974701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2114.378910] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2123.149578] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2143.714255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2143.720469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2143.745694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2148.233770] Lustre: server umount lustre-OST0000 complete [ 2153.953771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.981512] Lustre: Skipped 1 previous similar message [ 2154.003309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.006898] Lustre: Skipped 1 previous similar message [ 2159.075322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.084367] Lustre: Skipped 1 previous similar message [ 2163.170375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.180384] Lustre: Skipped 1 previous similar message [ 2167.264169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2167.523630] Lustre: server umount lustre-MDT0000 complete [ 2168.295521] LustreError: 63457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.320780] LustreError: 63457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2171.630926] LustreError: 63442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784851 with bad export cookie 15537796831173873604 [ 2171.639862] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.158643] Lustre: server umount lustre-MDT0001 complete [ 2184.624616] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2187.433238] Key type lgssc unregistered [ 2187.820987] LNet: 67024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.829105] LNetError: 67024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.855921] LNet: Removed LNI 192.168.202.134@tcp [ 2188.814349] Key type .llcrypt unregistered [ 2188.817304] Key type ._llcrypt unregistered [ 2218.341841] Key type ._llcrypt registered [ 2218.344550] Key type .llcrypt registered [ 2218.465655] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2219.765820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.923600] alg: No test for adler32 (adler32-zlib) [ 2221.053842] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2221.307500] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2223.065824] Key type lgssc registered [ 2224.564817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2234.205240] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:55:12 (1787784912) [ 2241.502592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2241.520400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.989410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.368850] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2253.284573] LustreError: 68731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.313344] LustreError: 68731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2257.162690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.638716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.383901] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2271.122884] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.536284] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2284.802066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2285.375726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.457471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2292.877667] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2301.607110] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2307.259316] Lustre: Failing over lustre-MDT0000 [ 2307.884817] Lustre: server umount lustre-MDT0000 complete [ 2308.080606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.099715] LustreError: 68726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.102462] Lustre: Skipped 1 previous similar message [ 2308.148059] LustreError: 68726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2313.192767] LustreError: 69682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.220990] LustreError: 69682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2318.310239] LustreError: 69458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.328305] LustreError: 69458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2319.130976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.353851] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.640579] Lustre: *** cfs_fail_loc=135, val=0*** [ 2319.779840] LustreError: 71819:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2319.786551] LustreError: 71819:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -2 [ 2319.796666] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2319.796666] [ 2319.810261] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2319.823582] LustreError: 71807:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2319.830659] LustreError: 71807:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2319.841794] LustreError: 71807:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2319.851304] LustreError: 71807:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2319.867231] Lustre: server umount lustre-MDT0000 complete [ 2319.870749] LustreError: 71807:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2323.431782] LustreError: 68730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.472077] LustreError: 68730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2327.612624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.708645] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.713786] LustreError: 68722:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2327.717953] LustreError: 72165:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.134@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c19a5383) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2328.080295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.163804] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2330.210492] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2333.173381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2333.215705] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2333.261325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2333.445237] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2343.400350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.406939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.442240] Lustre: Skipped 2 previous similar messages [ 2348.517131] LustreError: 70727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.529815] LustreError: 70727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2348.653178] Lustre: server umount lustre-OST0000 complete [ 2353.636554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2353.640381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.641863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.641870] Lustre: Skipped 1 previous similar message [ 2353.687562] Lustre: Skipped 1 previous similar message [ 2358.754446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.769629] Lustre: Skipped 2 previous similar messages [ 2359.824922] Lustre: server umount lustre-MDT0000 complete [ 2363.947469] LustreError: 68710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785044 with bad export cookie 5885813732750598708 [ 2363.970645] LustreError: 68710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2363.977959] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2364.586645] Lustre: server umount lustre-MDT0001 complete [ 2378.499473] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2382.991097] Key type lgssc unregistered [ 2383.453030] LNet: 73602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.463131] LNetError: 73602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.483805] LNet: Removed LNI 192.168.202.134@tcp [ 2384.789827] Key type .llcrypt unregistered [ 2384.799241] Key type ._llcrypt unregistered [ 2411.859818] Key type ._llcrypt registered [ 2411.864750] Key type .llcrypt registered [ 2412.063705] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2413.793934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.862777] alg: No test for adler32 (adler32-zlib) [ 2414.946636] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2415.210194] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2416.842753] Key type lgssc registered [ 2418.144101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2427.000250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2427.015488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.588599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.053314] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2438.118618] LustreError: 75017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.135612] LustreError: 75017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2441.569806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.134678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2446.627670] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2453.879408] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.990591] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2466.158993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.558419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.312846] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2473.720812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2481.418917] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2486.900054] Lustre: Failing over lustre-MDT0000 [ 2487.368801] Lustre: server umount lustre-MDT0000 complete [ 2488.290137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2488.295348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.315119] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.315789] LustreError: 75017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.338360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.354024] LustreError: 75017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2493.547345] LustreError: 76269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.552756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.768791] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.056578] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2498.222667] LustreError: 78104:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2498.232425] LustreError: 78104:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -9 [ 2498.255032] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2498.255032] [ 2498.279953] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2498.299806] LustreError: 78092:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2498.319782] LustreError: 78092:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2498.347202] LustreError: 78092:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2498.361919] LustreError: 78092:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2498.406964] Lustre: server umount lustre-MDT0000 complete [ 2498.421358] LustreError: 78092:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2498.660542] LustreError: 76269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.699186] LustreError: 76269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2506.243937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.364255] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.378104] LustreError: 75010:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2506.620971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.687669] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2508.893549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2510.989353] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2511.860569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2511.897159] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2512.004196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2522.096395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.114710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.124846] Lustre: Skipped 2 previous similar messages [ 2522.146993] Lustre: Skipped 1 previous similar message [ 2527.213442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.242913] Lustre: Skipped 1 previous similar message [ 2532.340732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2532.355353] Lustre: Skipped 1 previous similar message [ 2534.369545] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2534.624099] Lustre: server umount lustre-OST0000 complete [ 2542.570157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2542.585525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.594718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2542.606709] Lustre: Skipped 1 previous similar message [ 2544.820815] Lustre: server umount lustre-MDT0000 complete [ 2547.686813] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.714473] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2548.750738] LustreError: 77596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785228 with bad export cookie 14095553145399087321 [ 2548.753119] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2548.760841] LustreError: 77596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2549.006602] Lustre: server umount lustre-MDT0001 complete [ 2561.412601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2565.179833] Key type lgssc unregistered [ 2565.490567] LNet: 79891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.514631] LNetError: 79891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.546085] LNet: Removed LNI 192.168.202.134@tcp [ 2566.642938] Key type .llcrypt unregistered [ 2566.648558] Key type ._llcrypt unregistered [ 2581.486307] Key type ._llcrypt registered [ 2581.491599] Key type .llcrypt registered [ 2581.633666] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2581.980414] Key type .llcrypt unregistered [ 2581.985281] Key type ._llcrypt unregistered [ 2607.850436] Key type ._llcrypt registered [ 2607.859260] Key type .llcrypt registered [ 2608.008629] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2609.027038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.072779] alg: No test for adler32 (adler32-zlib) [ 2610.199323] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2610.398228] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2612.024168] Key type lgssc registered [ 2613.249375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.322470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2621.345437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.905431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.237854] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2632.165128] LustreError: 81813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.178723] LustreError: 81813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2637.286399] LustreError: 81812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.183570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.930413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2644.409475] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2653.637931] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2659.726633] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2667.164940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2667.503380] LustreError: 83632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.658485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.140761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2674.447251] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2682.222752] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2693.611308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.616652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2693.617199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2693.620133] Lustre: Skipped 1 previous similar message [ 2697.443508] Lustre: server umount lustre-OST0000 complete [ 2703.845499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.846994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.869463] Lustre: Skipped 1 previous similar message [ 2703.898694] Lustre: Skipped 2 previous similar messages [ 2708.163881] Lustre: server umount lustre-MDT0000 complete [ 2708.961430] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.994242] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2712.846549] LustreError: 81792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785392 with bad export cookie 8733517989794190361 [ 2712.863427] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.562827] Lustre: server umount lustre-MDT0001 complete [ 2725.082398] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2728.460280] Key type lgssc unregistered [ 2728.752156] LNet: 85349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.758705] LNetError: 85349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.832195] LNet: Removed LNI 192.168.202.134@tcp [ 2730.944167] Key type .llcrypt unregistered [ 2730.950137] Key type ._llcrypt unregistered [ 2757.679149] Key type ._llcrypt registered [ 2757.680717] Key type .llcrypt registered [ 2757.825042] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2759.332458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.424290] alg: No test for adler32 (adler32-zlib) [ 2760.657727] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2760.940391] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2762.688248] Key type lgssc registered [ 2763.933662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2773.102846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2773.120464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.765566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.766361] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2784.227584] LustreError: 86759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.255451] LustreError: 86759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2789.351532] LustreError: 86758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.536605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.001785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.940868] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2803.087543] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2808.782818] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2815.617415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.905384] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2816.005875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.092129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2823.539300] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2831.728748] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2839.521551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.530754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.545349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.548383] Lustre: Skipped 1 previous similar message [ 2840.871776] Lustre: server umount lustre-OST0000 complete [ 2846.177213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2846.184551] LustreError: Skipped 1 previous similar message [ 2846.188160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.212235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.214911] Lustre: Skipped 1 previous similar message [ 2849.763363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.764657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.798530] Lustre: Skipped 1 previous similar message [ 2851.263802] Lustre: server umount lustre-MDT0000 complete [ 2854.889819] LustreError: 87763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.907981] LustreError: 87763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2855.856293] LustreError: 86739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785535 with bad export cookie 10340724679763542957 [ 2855.877409] LustreError: 86739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2855.882199] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.385362] Lustre: server umount lustre-MDT0001 complete [ 2866.436907] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2869.184509] Key type lgssc unregistered [ 2869.566769] LNet: 90294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2869.589541] LNetError: 90294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2869.617868] LNet: Removed LNI 192.168.202.134@tcp [ 2870.732231] Key type .llcrypt unregistered [ 2870.737319] Key type ._llcrypt unregistered [ 2896.235558] Key type ._llcrypt registered [ 2896.238562] Key type .llcrypt registered [ 2896.356823] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2897.825709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2898.250257] alg: No test for adler32 (adler32-zlib) [ 2899.504802] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2899.833844] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2901.560258] Key type lgssc registered [ 2902.946314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2911.527181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2911.538842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.043032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.423752] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2923.494909] LustreError: 91713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.513818] LustreError: 91713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2927.035294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.361067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2932.763291] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2941.059763] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2947.406317] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2955.729128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.231894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.299749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2964.978658] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2974.455837] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2987.395429] Lustre: server umount lustre-OST0000 complete [ 2993.124355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.131852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.139567] Lustre: Skipped 1 previous similar message [ 2993.150264] Lustre: Skipped 1 previous similar message [ 2998.242177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.247048] Lustre: Skipped 1 previous similar message [ 3000.467546] Lustre: server umount lustre-MDT0000 complete [ 3003.364873] LustreError: 91710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.382046] LustreError: 91710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3004.088901] LustreError: 91695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785684 with bad export cookie 12829697727308156138 [ 3004.103449] LustreError: 91695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3004.105318] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.829730] Lustre: server umount lustre-MDT0001 complete [ 3028.172977] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3031.614969] Key type lgssc unregistered [ 3031.939850] LNet: 95252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3031.943758] LNetError: 95252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3033.011784] LNet: Removed LNI 192.168.202.134@tcp [ 3034.011409] Key type .llcrypt unregistered [ 3034.012778] Key type ._llcrypt unregistered [ 3061.932306] Key type ._llcrypt registered [ 3061.936316] Key type .llcrypt registered [ 3062.129519] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3063.920560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3063.978224] alg: No test for adler32 (adler32-zlib) [ 3065.145452] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3065.488539] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3067.248156] Key type lgssc registered [ 3068.307947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3076.801850] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3076.822597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.458066] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3097.568355] LustreError: 96639:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 3124.256758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3131.199842] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3139.362176] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.816448] Lustre: server umount lustre-OST0000 complete [ 3152.976892] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:10:31 (1787785831) [ 3161.213526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.591589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.533972] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3166.689161] LustreError: 98278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.706958] LustreError: 98278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3171.813176] LustreError: 98279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.762571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.118202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3180.383786] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3188.301530] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3194.742835] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3202.911927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3203.410677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3206.541057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3212.065078] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3221.388539] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3236.321294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3236.324647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.334635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3236.350990] Lustre: Skipped 1 previous similar message [ 3236.372517] Lustre: Skipped 1 previous similar message [ 3240.655336] Lustre: server umount lustre-OST0000 complete [ 3246.567733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.585499] Lustre: Skipped 1 previous similar message [ 3246.587031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.608502] Lustre: Skipped 1 previous similar message [ 3251.689642] LustreError: 98273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.702961] LustreError: 98273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3251.729639] Lustre: server umount lustre-MDT0000 complete [ 3255.402643] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785935 with bad export cookie 13147518835092200778 [ 3255.404620] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.423081] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3256.059802] Lustre: server umount lustre-MDT0001 complete [ 3266.969292] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3269.691150] Key type lgssc unregistered [ 3269.993879] LNet: 101978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3270.004026] LNetError: 101978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3270.020961] LNet: Removed LNI 192.168.202.134@tcp [ 3270.983792] Key type .llcrypt unregistered [ 3270.988815] Key type ._llcrypt unregistered [ 3315.833603] Key type ._llcrypt registered [ 3315.842648] Key type .llcrypt registered [ 3315.951304] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3317.463644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3317.483210] alg: No test for adler32 (adler32-zlib) [ 3318.633468] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3318.837202] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3320.496829] Key type lgssc registered [ 3321.678660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3328.198511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3328.214299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.685362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.403432] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3339.236741] LustreError: 104158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.255996] LustreError: 104158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3343.185897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.805247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3349.141487] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3357.803749] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3364.013151] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3371.199140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3371.578270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.650724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3378.104826] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3385.515678] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3398.126376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.130930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.143020] Lustre: Skipped 1 previous similar message [ 3398.160913] Lustre: Skipped 1 previous similar message [ 3401.583174] Lustre: server umount lustre-OST0000 complete [ 3408.364195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.367499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.392091] Lustre: Skipped 1 previous similar message [ 3408.410398] Lustre: Skipped 1 previous similar message [ 3411.678780] Lustre: server umount lustre-MDT0000 complete [ 3413.477870] LustreError: 106159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.516577] LustreError: 106159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3416.097191] LustreError: 104140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786096 with bad export cookie 3564104873803306724 [ 3416.105046] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.115061] LustreError: 104140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3416.473208] Lustre: server umount lustre-MDT0001 complete [ 3429.562495] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3432.617541] Key type lgssc unregistered [ 3432.969944] LNet: 107695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3432.981338] LNetError: 107695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3433.010655] LNet: Removed LNI 192.168.202.134@tcp [ 3434.071540] Key type .llcrypt unregistered [ 3434.073874] Key type ._llcrypt unregistered [ 3453.899095] Key type ._llcrypt registered [ 3453.907089] Key type .llcrypt registered [ 3454.053672] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3455.378024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3455.449099] alg: No test for adler32 (adler32-zlib) [ 3456.474797] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3456.664797] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3458.312671] Key type lgssc registered [ 3459.588792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3466.964347] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3466.998169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.592907] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3487.712428] LustreError: 108939:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 3514.400225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.912850] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3527.861685] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3533.918442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.066727] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3534.073981] LustreError: 109961:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3534.077082] LustreError: 109961:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3534.079877] LustreError: 109961:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3534.118756] Lustre: server umount lustre-MDT0000 complete [ 3534.122648] LustreError: 109961:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3553.840098] Lustre: server umount lustre-OST0000 complete [ 3564.188267] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 3575.041607] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3612.116589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.342948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3612.359901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3612.416717] Lustre: lustre-MDT0000: new disk, initializing [ 3612.477669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.491538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3616.071758] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3626.519452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.706422] Lustre: 112771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3626.786349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3626.790481] Lustre: Skipped 1 previous similar message [ 3626.890711] Lustre: lustre-MDT0001: new disk, initializing [ 3627.079448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3627.201645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3627.206247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3633.476461] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3646.854557] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3653.630640] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3662.743802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.947438] Lustre: lustre-OST0000: new disk, initializing [ 3662.951142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3662.954308] Lustre: 114742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3663.021712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.461486] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3670.674606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3670.690533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3670.923740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3681.528541] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3686.518817] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3686.742966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3691.912168] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3692.140505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3696.615332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.630925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.655310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.659224] Lustre: Skipped 1 previous similar message [ 3701.738871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.752857] Lustre: Skipped 1 previous similar message [ 3706.857439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.866473] Lustre: Skipped 1 previous similar message [ 3708.385474] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3708.738889] Lustre: server umount lustre-OST0000 complete [ 3714.029538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3714.037279] LustreError: Skipped 1 previous similar message [ 3714.041323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.056043] Lustre: Skipped 1 previous similar message [ 3714.064985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.100841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.612669] Lustre: server umount lustre-MDT0000 complete [ 3722.277145] LustreError: 113566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786402 with bad export cookie 4540656826493491707 [ 3722.278273] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.292214] LustreError: 113566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.689650] Lustre: server umount lustre-MDT0001 complete [ 3733.187669] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:20:11 (1787786411) [ 3737.355356] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3753.952782] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 3765.583303] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3801.416075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.717274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.751279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3801.849880] Lustre: lustre-MDT0000: new disk, initializing [ 3801.926393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.949657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3806.874397] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3818.272497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.405151] Lustre: 119646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3818.464710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3818.471821] Lustre: Skipped 1 previous similar message [ 3818.549279] Lustre: lustre-MDT0001: new disk, initializing [ 3818.647928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3818.677323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3818.692122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3823.515144] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3833.364395] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3839.921215] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3847.045828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3847.397134] Lustre: lustre-OST0000: new disk, initializing [ 3847.399954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3847.409927] Lustre: 121612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3847.561630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3854.299702] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3854.414096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3854.426330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3854.542830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3863.701941] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3867.604062] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3867.776258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3872.275177] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3872.470369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3875.297442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.307897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.309086] LustreError: Skipped 1 previous similar message [ 3875.322882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.326469] Lustre: Skipped 1 previous similar message [ 3875.363995] Lustre: Skipped 3 previous similar messages [ 3881.302657] Lustre: server umount lustre-OST0000 complete [ 3890.661325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3890.664142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.677704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.677711] Lustre: Skipped 2 previous similar messages [ 3890.720249] Lustre: Skipped 1 previous similar message [ 3892.479601] Lustre: server umount lustre-MDT0000 complete [ 3895.791596] LustreError: 121619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.840994] LustreError: 121619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3896.423195] LustreError: 119638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786576 with bad export cookie 4540656826493492589 [ 3896.437533] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3896.441647] LustreError: 119638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3896.921307] Lustre: server umount lustre-MDT0001 complete [ 3906.127372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.724157] LustreError: 123830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.786617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.161196] LustreError: 123831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.673705] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3916.263272] LustreError: 123830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.379323] LustreError: 123831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.778535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.566534] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3934.915923] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3942.036953] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3949.225254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.155462] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3963.202900] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3973.838724] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3982.307237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3982.321125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.344760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3982.350617] Lustre: Skipped 1 previous similar message [ 3984.999768] Lustre: server umount lustre-OST0000 complete [ 3995.632498] Lustre: server umount lustre-MDT0000 complete [ 3998.179021] LustreError: 123831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.206199] LustreError: 123831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3999.599623] LustreError: 123811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786679 with bad export cookie 4540656826493493471 [ 3999.621581] LustreError: 123811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3999.622700] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3999.964430] Lustre: server umount lustre-MDT0001 complete [ 4014.490187] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4018.157598] Key type lgssc unregistered [ 4018.444946] LNet: 127466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4018.466347] LNetError: 127466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4018.485324] LNet: Removed LNI 192.168.202.134@tcp [ 4019.478123] Key type .llcrypt unregistered [ 4019.479861] Key type ._llcrypt unregistered [ 4038.493475] Key type ._llcrypt registered [ 4038.495583] Key type .llcrypt registered [ 4038.604812] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4057.302737] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4058.717495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4058.759213] alg: No test for adler32 (adler32-zlib) [ 4060.050373] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4060.271307] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4061.952172] Key type lgssc registered [ 4063.456581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4101.167736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4101.211669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.415485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4102.441719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4102.499547] Lustre: lustre-MDT0000: new disk, initializing [ 4102.608985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.624904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4107.760715] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4117.376371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.559732] Lustre: 130622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4117.646202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4117.656233] Lustre: Skipped 1 previous similar message [ 4117.733553] Lustre: lustre-MDT0001: new disk, initializing [ 4117.818835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.872968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4117.881230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4123.402451] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4136.946600] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.913497] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4151.615534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.878365] Lustre: lustre-OST0000: new disk, initializing [ 4151.881326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4151.886811] Lustre: 132592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4151.984552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.327193] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4161.075775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4161.084969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4161.126172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4168.604296] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4174.099424] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4174.364377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4179.670847] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4179.911684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4186.605407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4186.617034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.619046] LustreError: Skipped 1 previous similar message [ 4186.624243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.683243] Lustre: Skipped 1 previous similar message [ 4188.922424] Lustre: server umount lustre-OST0000 complete [ 4194.785589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4194.787955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.796514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.808357] Lustre: Skipped 1 previous similar message [ 4196.834137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.838250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.856089] Lustre: Skipped 1 previous similar message [ 4199.215938] Lustre: server umount lustre-MDT0000 complete [ 4202.462071] LustreError: 130613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786882 with bad export cookie 8890176200588614903 [ 4202.467321] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.474117] LustreError: 130613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4202.778642] Lustre: server umount lustre-MDT0001 complete [ 4212.326950] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:28:10 (1787786890) [ 4219.150813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.623595] LustreError: 135002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.650751] LustreError: 135002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4219.716576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.786722] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4224.998584] LustreError: 135003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.090857] LustreError: 135002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.205907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.218041] LustreError: 135003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.695642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4241.403975] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4252.343380] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4259.002699] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4264.417344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.418400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.431307] Lustre: Skipped 1 previous similar message [ 4264.457543] Lustre: Skipped 1 previous similar message [ 4269.541523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.553968] Lustre: Skipped 1 previous similar message [ 4276.193441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4276.481377] Lustre: server umount lustre-MDT0000 complete [ 4279.778725] LustreError: 136243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.806167] LustreError: 136243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4280.417102] LustreError: 134984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786960 with bad export cookie 8890176200588615778 [ 4280.429081] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4280.900941] Lustre: server umount lustre-MDT0001 complete [ 4289.257565] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:29:27 (1787786967) [ 4296.305302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.906429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4317.025073] LustreError: 137418:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 4342.752360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.915469] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4358.530878] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4360.745931] Lustre: server umount lustre-OST0000 complete [ 4370.411524] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:30:49 (1787787049) [ 4377.856111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.352255] LustreError: 138865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.421866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.252559] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4392.605756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.248390] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4406.260741] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4412.051322] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.807134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.188421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.197616] Lustre: Skipped 1 previous similar message [ 4427.992688] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4436.753245] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4454.370034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.374192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.394557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4454.407665] Lustre: Skipped 4 previous similar messages [ 4456.288439] Lustre: server umount lustre-MDT0000 complete [ 4459.499355] LustreError: 138865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4459.533752] LustreError: 138865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4460.435301] LustreError: 138845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787140 with bad export cookie 8890176200588616450 [ 4460.448340] LustreError: 138845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4460.461896] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4460.901578] Lustre: server umount lustre-MDT0001 complete [ 4474.304951] Lustre: server umount lustre-OST0000 complete [ 4484.331360] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:32:43 (1787787163) [ 4492.023923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.448334] LustreError: 142539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.519966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.088428] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4505.826768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.055250] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4520.503582] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4526.936816] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4534.281866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4537.906046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4541.546360] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4550.014557] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4555.739353] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4556.046742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4560.913967] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4561.167705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4563.431587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4563.439163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.454122] Lustre: Skipped 2 previous similar messages [ 4563.461499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4577.248432] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4577.541289] Lustre: server umount lustre-OST0000 complete [ 4581.858513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.878166] Lustre: Skipped 2 previous similar messages [ 4588.054822] Lustre: server umount lustre-MDT0000 complete [ 4592.104907] LustreError: 142535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.128440] LustreError: 142535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4592.785574] LustreError: 142520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787272 with bad export cookie 8890176200588617514 [ 4592.803576] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.804644] LustreError: 142520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4593.181473] Lustre: server umount lustre-MDT0001 complete [ 4602.694528] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:34:41 (1787787281) [ 4609.601673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.120755] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4630.240524] LustreError: 146700:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 4655.969153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.978489] Lustre: Skipped 2 previous similar messages [ 4662.684034] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4670.721201] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4677.017037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.557087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4682.133233] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4690.773991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.230926] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4704.920473] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4711.637415] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4716.080289] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4716.401348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4723.710948] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4724.130501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4727.270505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4727.271377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4727.306285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4727.306306] Lustre: Skipped 10 previous similar messages [ 4727.308185] Lustre: Skipped 2 previous similar messages [ 4727.380060] LustreError: Skipped 1 previous similar message [ 4733.098891] Lustre: server umount lustre-MDT0000 complete [ 4737.505642] LustreError: 147737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.514854] LustreError: 147737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4737.550878] LustreError: 146706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787417 with bad export cookie 8890176200588618543 [ 4737.554490] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4737.563212] LustreError: 146706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4738.099195] Lustre: server umount lustre-MDT0001 complete [ 4753.514912] Lustre: server umount lustre-OST0000 complete [ 4764.619499] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:37:22 (1787787442) [ 4772.479778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4788.000650] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4793.120528] LustreError: 150898:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 4825.516743] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4834.336541] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4841.730202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.524339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4847.490927] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4857.562068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.271485] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4872.985541] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4879.701610] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4887.769640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.015993] Lustre: lustre-OST0001: new disk, initializing [ 4888.019874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4888.025658] Lustre: 153781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4895.329533] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4895.780764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4895.788458] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4895.907727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4907.150786] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4912.682899] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4912.911658] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4917.726296] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4918.016310] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4921.841678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.850644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.861835] Lustre: Skipped 1 previous similar message [ 4921.873779] Lustre: Skipped 5 previous similar messages [ 4926.344338] Lustre: server umount lustre-OST0000 complete [ 4931.554615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4937.670638] Lustre: server umount lustre-OST0001 complete [ 4948.696618] Lustre: server umount lustre-MDT0000 complete [ 4953.513762] LustreError: 150905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787633 with bad export cookie 8890176200588619495 [ 4953.517612] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.524424] LustreError: 150905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4953.959492] Lustre: server umount lustre-MDT0001 complete [ 4985.595954] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:41:04 (1787787664) [ 4987.206780] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4988.962150] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4990.784283] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:41:09 (1787787669) [ 4997.754062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.241984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4998.308658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.512755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.520104] Lustre: Skipped 6 previous similar messages [ 5002.155250] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5010.565525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.677491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5010.725601] Lustre: 157344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5015.399188] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5022.033978] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5027.399308] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5034.891995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5035.034713] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5042.753483] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5044.346083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5051.474734] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5057.727685] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5057.934242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5063.369661] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5063.762922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5069.804419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.813045] LustreError: Skipped 2 previous similar messages [ 5069.815810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.829595] Lustre: Skipped 4 previous similar messages [ 5069.833491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5069.837207] Lustre: Skipped 9 previous similar messages [ 5072.249501] Lustre: server umount lustre-OST0000 complete [ 5095.549804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.029697] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.049710] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5102.453931] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5110.197680] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5110.249044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.286482] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5115.540172] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5115.740897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5120.731986] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5121.015993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5126.894898] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b9811011000.ost_server_uuid 50 [ 5128.526892] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b9811011000.ost_server_uuid in FULL state after 0 sec [ 5128.798557] LustreError: 157351:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.34@tcp) returned error from blocking AST (req@00000000c2bbcff2 x1874630268957568 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000007f679e4a/0x7b604026fab77841 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.34@tcp remote: 0x207adcb6f76f5cb2 expref: 5 pid: 159455 timeout: 5228 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5128.856671] LustreError: lustre-MDT0000: A client on nid 192.168.202.34@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5128.869715] LustreError: 157339:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.34@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000007f679e4a/0x7b604026fab77841 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.34@tcp remote: 0x207adcb6f76f5cb2 expref: 6 pid: 159455 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5134.823153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5139.721563] Lustre: server umount lustre-OST0000 complete [ 5149.965809] Lustre: server umount lustre-MDT0000 complete [ 5151.203974] LustreError: 158175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.225724] LustreError: 158175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 5153.694039] LustreError: 157334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787833 with bad export cookie 8890176200588620678 [ 5153.698278] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.708799] LustreError: 157334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5160.140124] Lustre: server umount lustre-MDT0001 complete [ 5173.135235] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5176.423491] Key type lgssc unregistered [ 5176.796666] LNet: 163459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5176.816730] LNetError: 163459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5176.842267] LNet: Removed LNI 192.168.202.134@tcp [ 5177.816939] Key type .llcrypt unregistered [ 5177.819246] Key type ._llcrypt unregistered [ 5204.168109] Key type ._llcrypt registered [ 5204.169456] Key type .llcrypt registered [ 5204.245680] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5204.958648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5205.089389] alg: No test for adler32 (adler32-zlib) [ 5206.239643] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 5206.443920] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5208.105945] Key type lgssc registered [ 5209.268848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5217.250784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5217.259148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.785868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.044422] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5228.005806] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.046255] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5233.123850] LustreError: 164876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.625680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.078137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5238.972990] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5246.560635] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5252.221929] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5258.947158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.338685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.420290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5266.544624] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5275.843484] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5283.113623] Lustre: Failing over lustre-MDT0000 [ 5283.810939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.813063] LustreError: 165608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.828471] Lustre: Skipped 2 previous similar messages [ 5283.863987] LustreError: 165608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5285.443635] Lustre: server umount lustre-MDT0000 complete [ 5288.931788] LustreError: 164871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.962244] LustreError: 164871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5294.049524] LustreError: 164870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.075689] LustreError: 164870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5297.966638] LustreError: 164857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.34@tcp arrived at 1787787978 with bad export cookie 16352007974901705236 [ 5297.998087] LustreError: 164857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5304.290803] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.323101] LustreError: 164875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5305.312240] Lustre: 164187:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787969/real 1787787969] req@0000000085146943 x1874631470379648/t0(0) o400->MGC192.168.202.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787787985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5305.351248] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5305.438846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.563368] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe2ee009dd45eea76 [ 5315.587880] Lustre: MGC192.168.202.134@tcp: Connection restored to 0@lo (at 0@lo) [ 5315.959542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.019199] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5321.209376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5321.212904] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5321.420868] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5324.045693] Lustre: lustre-MDT0000: Denying connection for new client a02ce1ae-b328-40a6-85f9-23312f0937b9 (at 192.168.202.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5336.550307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.565729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5338.625831] Lustre: server umount lustre-OST0000 complete [ 5342.589048] LustreError: 168611:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5342.595729] Lustre: 167959:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5342.602410] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5342.617489] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5342.625225] LustreError: 167959:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000690eedf6 x1874631470400000/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5342.641428] LustreError: 167959:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5342.648722] LustreError: 167959:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5342.656233] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5343.073640] Lustre: server umount lustre-MDT0000 complete [ 5346.786272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.821361] Lustre: Skipped 1 previous similar message [ 5346.836105] LustreError: 164870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.853959] LustreError: 164870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5347.485413] LustreError: 164855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788027 with bad export cookie 16352007974901705334 [ 5347.487432] LustreError: 164184:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5347.493470] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5347.506957] LustreError: 164855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5350.334500] Lustre: server umount lustre-MDT0001 complete [ 5363.033615] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5366.443684] Key type lgssc unregistered [ 5366.716774] LNet: 169367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5366.721973] LNetError: 169367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5366.734580] LNet: Removed LNI 192.168.202.134@tcp [ 5367.800967] Key type .llcrypt unregistered [ 5367.803154] Key type ._llcrypt unregistered [ 5392.979284] Key type ._llcrypt registered [ 5392.982727] Key type .llcrypt registered [ 5393.102859] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5394.616657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5394.650686] alg: No test for adler32 (adler32-zlib) [ 5395.937337] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 5396.165775] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5397.816168] Key type lgssc registered [ 5399.106757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5406.953143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5406.977158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.575410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.769030] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5418.020738] LustreError: 170783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.039127] LustreError: 170783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5423.077207] LustreError: 170784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.692835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.271061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5428.970543] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5436.648924] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5442.173793] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5448.977643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5449.148089] LustreError: 172603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.362252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.498642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5455.722877] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5463.048738] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5469.153138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.156599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.167145] LustreError: Skipped 1 previous similar message [ 5469.170383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.187042] Lustre: Skipped 1 previous similar message [ 5469.205232] Lustre: Skipped 1 previous similar message [ 5474.280896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.288994] Lustre: Skipped 1 previous similar message [ 5474.481910] Lustre: server umount lustre-OST0000 complete [ 5479.398030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.409653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.416462] Lustre: Skipped 1 previous similar message [ 5484.513462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5489.636406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5489.646516] Lustre: Skipped 2 previous similar messages [ 5492.192349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5492.496187] Lustre: server umount lustre-MDT0000 complete [ 5494.754503] LustreError: 171512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.776168] LustreError: 171512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5496.954227] LustreError: 170763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788177 with bad export cookie 1199170904289767687 [ 5496.976056] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.402778] Lustre: server umount lustre-MDT0001 complete [ 5510.135268] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5513.251114] Key type lgssc unregistered [ 5513.502665] LNet: 174319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5513.511242] LNetError: 174319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.542891] LNet: Removed LNI 192.168.202.134@tcp [ 5514.532344] Key type .llcrypt unregistered [ 5514.537892] Key type ._llcrypt unregistered [ 5553.782575] Key type ._llcrypt registered [ 5553.786470] Key type .llcrypt registered [ 5553.964968] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5555.260801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5555.290194] alg: No test for adler32 (adler32-zlib) [ 5556.318186] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 5556.524214] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5558.184155] Key type lgssc registered [ 5559.377788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5566.777761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5566.788674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.106899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.372198] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5578.212753] LustreError: 176023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.225620] LustreError: 176023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5580.183519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.670514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5584.448355] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5591.974364] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5597.289903] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5603.506211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.914376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.536790] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5613.237984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5618.097864] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5625.829083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5625.874214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.888697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5627.254406] Lustre: server umount lustre-OST0000 complete [ 5631.971413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5631.977954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.994333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.018283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.023899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.138629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5639.147385] Lustre: Skipped 2 previous similar messages [ 5644.257684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5644.266351] Lustre: Skipped 1 previous similar message [ 5645.280359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.589165] Lustre: server umount lustre-MDT0000 complete [ 5649.194688] LustreError: 176003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788329 with bad export cookie 3912447646226034616 [ 5649.204351] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5649.213047] LustreError: 176003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5649.747075] Lustre: server umount lustre-MDT0001 complete [ 5663.105454] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5665.959569] Key type lgssc unregistered [ 5666.345715] LNet: 179558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5666.354417] LNetError: 179558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5666.373212] LNet: Removed LNI 192.168.202.134@tcp [ 5667.423351] Key type .llcrypt unregistered [ 5667.427186] Key type ._llcrypt unregistered [ 5691.402446] Key type ._llcrypt registered [ 5691.407176] Key type .llcrypt registered [ 5691.543376] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5693.432328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5693.450373] alg: No test for adler32 (adler32-zlib) [ 5694.716433] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 5695.030401] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5696.703742] Key type lgssc registered [ 5698.057599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5707.490666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5707.527432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.162459] Lustre: *** cfs_fail_loc=135, val=0*** [ 5709.366349] LustreError: 181089:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5709.381835] LustreError: 181089:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -2 [ 5709.397092] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5709.397092] [ 5709.413100] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5709.424587] LustreError: 181050:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5709.430866] LustreError: 181050:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5709.440793] LustreError: 181050:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5709.447310] LustreError: 181050:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5709.503571] Lustre: server umount lustre-MDT0000 complete [ 5709.506566] LustreError: 181050:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5716.101092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.650732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.060298] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5726.187523] LustreError: 181460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.217090] LustreError: 181460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5731.266759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.297942] LustreError: 181461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5731.569174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.033908] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5744.742144] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5750.392848] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5756.386246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.408665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.415663] Lustre: Skipped 1 previous similar message [ 5757.473959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.481196] Lustre: Skipped 1 previous similar message [ 5759.047206] Lustre: server umount lustre-MDT0000 complete [ 5762.531424] LustreError: 181455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5762.564377] LustreError: 181455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5763.038601] LustreError: 181442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788443 with bad export cookie 13073641768495780541 [ 5763.052496] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.072922] LustreError: 181442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5763.477849] Lustre: server umount lustre-MDT0001 complete [ 5774.618228] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5777.620429] Key type lgssc unregistered [ 5777.921782] LNet: 183785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5777.931029] LNetError: 183785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5777.944826] LNet: Removed LNI 192.168.202.134@tcp [ 5778.677919] Key type .llcrypt unregistered [ 5778.683179] Key type ._llcrypt unregistered [ 5801.664513] Key type ._llcrypt registered [ 5801.666218] Key type .llcrypt registered [ 5801.753475] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5801.929621] Key type .llcrypt unregistered [ 5801.931216] Key type ._llcrypt unregistered [ 5822.609460] Key type ._llcrypt registered [ 5822.616325] Key type .llcrypt registered [ 5822.771694] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5824.639414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5824.684928] alg: No test for adler32 (adler32-zlib) [ 5825.723625] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 5825.963518] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5827.794085] Key type lgssc registered [ 5828.926280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5836.204128] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5836.225444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5852.064400] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5857.184359] LustreError: 185921:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 5883.808411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5891.281376] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5898.645490] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5904.868081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.365863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.430315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5910.240512] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5910.500285] LustreError: 185947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.525422] LustreError: 185947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5915.621784] LustreError: 185946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.694489] LustreError: 186959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.732719] LustreError: 186959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5919.228491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.580274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5924.317478] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5932.166065] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5937.578960] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5940.469498] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5951.457725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5951.467091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.482428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5955.573456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5955.579295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5955.587181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.632764] Lustre: Skipped 1 previous similar message [ 5957.518595] Lustre: server umount lustre-OST0000 complete [ 5965.794080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.798236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5965.798332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5965.808815] Lustre: Skipped 1 previous similar message [ 5968.306469] Lustre: server umount lustre-MDT0000 complete [ 5970.913110] LustreError: 186958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.949955] LustreError: 186958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5971.812555] LustreError: 185928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788651 with bad export cookie 4505472353856397632 [ 5971.821447] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.821796] LustreError: 185928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5972.200606] Lustre: server umount lustre-MDT0001 complete [ 5985.437504] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5988.468319] Key type lgssc unregistered [ 5988.725263] LNet: 189887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5988.733346] LNetError: 189887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5988.752090] LNet: Removed LNI 192.168.202.134@tcp [ 5989.697764] Key type .llcrypt unregistered [ 5989.699986] Key type ._llcrypt unregistered [ 6013.151489] Key type ._llcrypt registered [ 6013.154571] Key type .llcrypt registered [ 6013.260773] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6014.628135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6014.645759] alg: No test for adler32 (adler32-zlib) [ 6015.736474] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6016.035536] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6017.696161] Key type lgssc registered [ 6018.727022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.524579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.554494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.135162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.247278] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6038.501027] LustreError: 191303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.517512] LustreError: 191303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6041.326211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.752277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6046.977709] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6055.731390] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6061.614794] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6067.772781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6068.235417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.338753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6074.688357] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6082.262160] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6088.294344] Lustre: Failing over lustre-MDT0000 [ 6088.571800] Lustre: server umount lustre-MDT0000 complete [ 6090.362568] LustreError: 191298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.749438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.752867] LustreError: 191303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.775512] Lustre: Skipped 1 previous similar message [ 6091.801703] LustreError: 191303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6095.469518] LustreError: 191298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.584834] LustreError: 191298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.607394] LustreError: 191298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6107.105659] Lustre: 190617:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787788771/real 1787788771] req@00000000bc478a87 x1874632319723136/t0(0) o400->MGC192.168.202.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787788787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6107.130928] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.133443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.811530] LustreError: 192542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.825559] LustreError: 192542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6117.349123] LustreError: 190613:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000190e3d13 x1874632319727872/t0(0) o250->MGC192.168.202.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6117.803380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.906464] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6121.053457] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6123.002920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6123.067772] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6123.134752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6123.338164] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6127.747183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6131.300903] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6142.947291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6142.955089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.961151] LustreError: Skipped 1 previous similar message [ 6142.965711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6142.994364] Lustre: Skipped 2 previous similar messages [ 6143.034397] Lustre: Skipped 1 previous similar message [ 6143.489978] Lustre: server umount lustre-OST0000 complete [ 6148.578618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6148.579405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.600591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6148.608427] Lustre: Skipped 1 previous similar message [ 6153.699654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6153.714953] Lustre: Skipped 1 previous similar message [ 6158.818747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.828611] Lustre: Skipped 1 previous similar message [ 6161.376154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6161.542340] Lustre: server umount lustre-MDT0000 complete [ 6163.939899] LustreError: 194604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.958768] LustreError: 194604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6165.461937] LustreError: 191285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788845 with bad export cookie 17697303277229953302 [ 6165.473497] LustreError: 191285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6165.476166] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.903745] Lustre: server umount lustre-MDT0001 complete [ 6178.079245] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6181.598600] Key type lgssc unregistered [ 6182.021383] LNet: 196113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6182.030382] LNetError: 196113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6182.061143] LNet: Removed LNI 192.168.202.134@tcp [ 6183.000372] Key type .llcrypt unregistered [ 6183.004526] Key type ._llcrypt unregistered [ 6208.563345] Key type ._llcrypt registered [ 6208.567791] Key type .llcrypt registered [ 6208.845319] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6210.504313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6210.596162] alg: No test for adler32 (adler32-zlib) [ 6211.877359] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6212.126576] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6213.792454] Key type lgssc registered [ 6214.692168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6222.825597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6222.863552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.679678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.480385] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6234.086290] LustreError: 197527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.113232] LustreError: 197527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6238.984750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.207797] LustreError: 197526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6239.696032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6246.190714] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6254.074690] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6260.062852] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6267.145413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6267.610481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.709371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6273.522833] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6280.409455] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6284.314257] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6291.899057] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6309.090895] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6325.221619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.223992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6325.224159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6325.232110] Lustre: Skipped 1 previous similar message [ 6325.256704] Lustre: Skipped 1 previous similar message [ 6328.546718] Lustre: server umount lustre-OST0000 complete [ 6335.458721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.460586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.477190] Lustre: Skipped 1 previous similar message [ 6340.579511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6340.585370] Lustre: Skipped 2 previous similar messages [ 6345.700116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6345.707551] Lustre: Skipped 1 previous similar message [ 6347.232201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6347.590359] Lustre: server umount lustre-MDT0000 complete [ 6350.820771] LustreError: 197523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.852483] LustreError: 197523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6351.199448] LustreError: 197508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789031 with bad export cookie 13149416207017499810 [ 6351.206083] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.208532] LustreError: 197508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6351.620825] Lustre: server umount lustre-MDT0001 complete [ 6363.233916] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6366.827076] Key type lgssc unregistered [ 6367.150348] LNet: 201424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6367.160460] LNetError: 201424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6368.178374] LNet: Removed LNI 192.168.202.134@tcp [ 6369.153600] Key type .llcrypt unregistered [ 6369.156901] Key type ._llcrypt unregistered [ 6391.009456] Key type ._llcrypt registered [ 6391.011915] Key type .llcrypt registered [ 6391.116895] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6392.239266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6392.422621] alg: No test for adler32 (adler32-zlib) [ 6393.583961] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6393.914062] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6395.600166] Key type lgssc registered [ 6396.834970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.870614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6404.882290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.528464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.987361] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6416.868364] LustreError: 202690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.878257] LustreError: 202690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6419.150670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.492512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6424.588530] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6432.779646] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6438.754230] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6445.601273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6446.066761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.206244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6452.823481] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6460.974363] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6467.957749] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6477.773804] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6488.707979] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6493.927259] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6505.445503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.449937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.466411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6505.491216] Lustre: Skipped 1 previous similar message [ 6508.845478] Lustre: server umount lustre-OST0000 complete [ 6515.691084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.701331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.705196] Lustre: Skipped 1 previous similar message [ 6515.723784] Lustre: Skipped 2 previous similar messages [ 6519.393828] Lustre: server umount lustre-MDT0000 complete [ 6520.804739] LustreError: 202690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.833665] LustreError: 202690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6522.990139] LustreError: 205366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789203 with bad export cookie 1904893086529996441 [ 6523.004304] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.010522] LustreError: 205366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6523.394397] Lustre: server umount lustre-MDT0001 complete [ 6535.323213] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6538.205847] Key type lgssc unregistered [ 6538.467900] LNet: 207568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6538.472483] LNetError: 207568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6538.489892] LNet: Removed LNI 192.168.202.134@tcp [ 6539.486921] Key type .llcrypt unregistered [ 6539.490230] Key type ._llcrypt unregistered [ 6565.856428] Key type ._llcrypt registered [ 6565.858861] Key type .llcrypt registered [ 6565.943390] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6567.293588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6567.385139] alg: No test for adler32 (adler32-zlib) [ 6568.587396] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6568.953915] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6570.688286] Key type lgssc registered [ 6571.764123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6580.027428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6580.041582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.313551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.732650] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6591.467887] LustreError: 208981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6591.476993] LustreError: 208981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6594.376025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.791846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6598.910504] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6606.941274] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.454231] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6621.781872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6622.473690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6626.579695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6631.428137] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6641.483445] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6657.506180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6657.523805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.547487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6660.579190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6660.584909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6660.617567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6663.126636] Lustre: server umount lustre-OST0000 complete [ 6670.817572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.823495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6670.830571] Lustre: Skipped 1 previous similar message [ 6670.842960] Lustre: Skipped 2 previous similar messages [ 6673.201518] Lustre: server umount lustre-MDT0000 complete [ 6675.948760] LustreError: 210706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6675.964134] LustreError: 210706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6676.873676] LustreError: 208963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789356 with bad export cookie 53650603951440606 [ 6676.877721] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6676.882786] LustreError: 208963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6677.227140] Lustre: server umount lustre-MDT0001 complete [ 6687.254246] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6689.445170] Key type lgssc unregistered [ 6689.751859] LNet: 212669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6689.759581] LNetError: 212669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6689.782988] LNet: Removed LNI 192.168.202.134@tcp [ 6690.609848] Key type .llcrypt unregistered [ 6690.611667] Key type ._llcrypt unregistered [ 6710.004317] Key type ._llcrypt registered [ 6710.008621] Key type .llcrypt registered [ 6710.117982] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6711.306094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6711.343435] alg: No test for adler32 (adler32-zlib) [ 6712.569303] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6712.786096] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6714.450104] Key type lgssc registered [ 6715.933856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6724.822790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6724.841574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.289315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.595460] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6736.870463] LustreError: 213939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.889069] LustreError: 213939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6739.937608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.534345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.071883] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6756.988187] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6762.458587] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.085259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.407068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6771.525666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6775.930286] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6783.347850] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6801.379894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6801.392837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.410571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6805.858274] Lustre: server umount lustre-OST0000 complete [ 6811.620327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.631891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6811.638899] Lustre: Skipped 2 previous similar messages [ 6811.641982] Lustre: Skipped 1 previous similar message [ 6815.739918] Lustre: server umount lustre-MDT0000 complete [ 6816.738941] LustreError: 213938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.748056] LustreError: 213938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6818.848761] LustreError: 213920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789498 with bad export cookie 4094193250463821375 [ 6818.855162] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6818.858686] LustreError: 213920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6819.331991] Lustre: server umount lustre-MDT0001 complete [ 6829.476342] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6832.103233] Key type lgssc unregistered [ 6832.461841] LNet: 217623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6832.471602] LNetError: 217623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6832.483622] LNet: Removed LNI 192.168.202.134@tcp [ 6833.306116] Key type .llcrypt unregistered [ 6833.307547] Key type ._llcrypt unregistered [ 6849.834407] Key type ._llcrypt registered [ 6849.840895] Key type .llcrypt registered [ 6849.988688] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6851.089865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6851.101642] alg: No test for adler32 (adler32-zlib) [ 6852.273575] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6852.470271] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6854.144174] Key type lgssc registered [ 6855.196788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6862.433504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6862.455920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.938923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6868.044689] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6874.082565] LustreError: 218868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6874.097212] LustreError: 218868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6875.404705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.782080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6879.864706] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6887.857922] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6894.863850] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6902.062293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.530929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.973793] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6909.710310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6914.523852] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.081896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6924.143290] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6924.360452] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.574498] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6938.410690] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6939.723937] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6951.120476] Lustre: Permanently deactivating lustre-OST0001 [ 6951.134496] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6957.585183] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6964.583511] Lustre: Permanently reactivating lustre-OST0001 [ 6964.596558] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6964.599866] Lustre: Skipped 2 previous similar messages [ 6965.899308] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.910173] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6965.914322] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6965.923674] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6965.961057] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6975.498621] Lustre: server umount lustre-OST0001 complete [ 6976.489093] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.490817] LustreError: 220688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.504115] Lustre: Skipped 2 previous similar messages [ 6976.530239] LustreError: 220688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6981.602246] LustreError: 222027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.604403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6981.609250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.610410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6981.616679] LustreError: 222027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6981.620845] LustreError: Skipped 1 previous similar message [ 6986.729086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.729786] LustreError: 220691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.739668] Lustre: Skipped 2 previous similar messages [ 6986.749559] LustreError: 220691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6991.845626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6991.848098] LustreError: 222027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.849294] Lustre: Skipped 1 previous similar message [ 6993.376307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6993.481376] Lustre: server umount lustre-OST0000 complete [ 6998.501412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6998.516771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.529531] Lustre: Skipped 1 previous similar message [ 6998.538917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7004.090254] Lustre: server umount lustre-MDT0000 complete [ 7007.202498] LustreError: 218868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.233295] LustreError: 218868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7007.387962] LustreError: 218850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789687 with bad export cookie 15259067785177565238 [ 7007.396110] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.402350] LustreError: 218850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7007.715047] Lustre: server umount lustre-MDT0001 complete [ 7018.084785] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7020.807909] Key type lgssc unregistered [ 7021.072846] LNet: 223923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7021.086385] LNetError: 223923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7021.115817] LNet: Removed LNI 192.168.202.134@tcp [ 7022.014572] Key type .llcrypt unregistered [ 7022.015743] Key type ._llcrypt unregistered [ 7057.929472] Key type ._llcrypt registered [ 7057.932287] Key type .llcrypt registered [ 7058.117133] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7059.429576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7059.471058] alg: No test for adler32 (adler32-zlib) [ 7060.582487] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7060.787070] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7062.481980] Key type lgssc registered [ 7063.559128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7070.817891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7070.840679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.185192] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7072.199388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7072.363875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.800403] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7083.924684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.026190] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7084.083910] Lustre: 226126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7084.424166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.475654] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7096.310686] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7101.743971] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7107.538452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7107.675138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7107.857497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.353919] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7116.914512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7119.699362] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7123.042158] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7130.441514] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7135.845776] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7146.466516] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7160.669797] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7160.676969] Lustre: Skipped 1 previous similar message [ 7183.522439] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7183.528132] Lustre: Skipped 2 previous similar messages [ 7217.475146] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7217.488281] Lustre: Skipped 3 previous similar messages [ 7291.328322] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7291.332674] Lustre: Skipped 8 previous similar messages [ 7301.091195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.099306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.118833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.607148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.614682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.629958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7303.988291] Lustre: server umount lustre-OST0000 complete [ 7309.795995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7309.803577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7309.816757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.842834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.856048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.807184] Lustre: server umount lustre-MDT0000 complete [ 7317.025980] LustreError: 226120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789997 with bad export cookie 4337522085762446578 [ 7317.029284] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.037859] LustreError: 226120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7317.317579] Lustre: server umount lustre-MDT0001 complete [ 7327.090947] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7330.051673] Key type lgssc unregistered [ 7330.304738] LNet: 230974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7330.321925] LNetError: 230974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7330.359390] LNet: Removed LNI 192.168.202.134@tcp [ 7331.206559] Key type .llcrypt unregistered [ 7331.208411] Key type ._llcrypt unregistered [ 7353.628887] Key type ._llcrypt registered [ 7353.632754] Key type .llcrypt registered [ 7353.795890] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7355.010682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7355.228162] alg: No test for adler32 (adler32-zlib) [ 7356.321852] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7356.630531] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7358.288550] Key type lgssc registered [ 7359.614725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7366.639083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7366.664708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7368.189842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.709851] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7378.405321] LustreError: 232388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.422278] LustreError: 232388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7381.319735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.588366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7385.783225] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7393.153923] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7398.412103] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7404.482464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7404.733480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.924190] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7412.966554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7420.159549] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.490755] Lustre: server umount lustre-OST0000 complete [ 7442.913789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.925252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.954814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.940140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.944291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.057613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7449.063446] Lustre: Skipped 2 previous similar messages [ 7453.154115] Lustre: 231697:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787790117/real 1787790117] req@0000000075731580 x1874633724823936/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787790133 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7453.183994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.204764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.213232] Lustre: Skipped 1 previous similar message [ 7454.176318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7454.240929] Lustre: 231699:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787790118/real 1787790118] req@0000000062f26624 x1874633724824704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787790134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7454.551797] Lustre: server umount lustre-MDT0000 complete [ 7457.808805] LustreError: 232369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790137 with bad export cookie 7498155881203165463 [ 7457.810701] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7457.826954] LustreError: 232369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7458.131366] Lustre: server umount lustre-MDT0001 complete [ 7467.659621] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7469.638907] Key type lgssc unregistered [ 7469.790796] LNet: 236079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7469.795555] LNetError: 236079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7470.821773] LNet: Removed LNI 192.168.202.134@tcp [ 7471.342813] Key type .llcrypt unregistered [ 7471.345225] Key type ._llcrypt unregistered [ 7492.480570] Key type ._llcrypt registered [ 7492.482204] Key type .llcrypt registered [ 7492.566408] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7493.359194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7493.486517] alg: No test for adler32 (adler32-zlib) [ 7494.506828] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7494.760758] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7496.448173] Key type lgssc registered [ 7497.659082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7767.267288] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7769.478377] Key type lgssc unregistered [ 7769.828860] LNet: 238085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7769.835343] LNetError: 238085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7769.850151] LNet: Removed LNI 192.168.202.134@tcp [ 7770.872673] Key type .llcrypt unregistered [ 7770.878575] Key type ._llcrypt unregistered [ 7801.852767] Key type ._llcrypt registered [ 7801.856449] Key type .llcrypt registered [ 7801.973243] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7820.229130] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7821.859225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7821.965254] alg: No test for adler32 (adler32-zlib) [ 7823.026855] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7823.247673] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7825.008673] Key type lgssc registered [ 7826.255911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7858.641693] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7610 sec ========== 20:28:57 (1787790537) [ 7860.469857] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:28:59 (1787790539) === [ 7864.342894] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:29:02 (1787790542) === [ 7905.450974] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7907.872413] Key type lgssc unregistered [ 7908.243076] LNet: 243327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7908.249930] LNetError: 243327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7908.269858] LNet: Removed LNI 192.168.202.134@tcp [ 7909.010698] Key type .llcrypt unregistered [ 7909.012925] Key type ._llcrypt unregistered