[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473359142 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003232] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007934] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010116] LSM: Security Framework initializing [ 0.011080] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021821] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026237] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] *** VALIDATE tmpfs *** [ 0.030469] *** VALIDATE proc *** [ 0.032166] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035133] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041713] debug: unmapping init [mem 0xffffffff90c59000-0xffffffff90c60fff] [ 0.043170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044658] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052307] rcu: Hierarchical SRCU implementation. [ 0.054390] smp: Bringing up secondary CPUs ... [ 0.055570] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059367] smp: Brought up 1 node, 4 CPUs [ 0.061009] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130605] node 0 deferred pages initialised in 66ms [ 0.134154] devtmpfs: initialized [ 0.135172] x86/mm: Memory block size: 128MB [ 0.137967] gcov: version magic: 0x41383552 [ 0.141265] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144293] pinctrl core: initialized pinctrl subsystem [ 0.146224] [ 0.146746] ************************************************************* [ 0.149011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151010] ** ** [ 0.154013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156007] ** ** [ 0.157010] ** This means that this kernel is built to expose internal ** [ 0.160013] ** IOMMU data structures, which may compromise security on ** [ 0.163011] ** your system. ** [ 0.165010] ** ** [ 0.167011] ** If you see this message and you are not debugging the ** [ 0.169011] ** kernel, report this immediately to your vendor! ** [ 0.171011] ** ** [ 0.172015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175011] ************************************************************* [ 0.178603] NET: Registered protocol family 16 [ 0.180439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189117] cpuidle: using governor menu [ 0.191560] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193573] PCI: Using configuration type 1 for base access [ 0.196121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205074] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208056] cryptd: max_cpu_qlen set to 1000 [ 0.210143] ACPI: Added _OSI(Module Device) [ 0.212013] ACPI: Added _OSI(Processor Device) [ 0.213012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218844] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226339] ACPI: Interpreter enabled [ 0.227058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229012] ACPI: Using IOAPIC for interrupt routing [ 0.231086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235324] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244834] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259526] acpiphp: Slot [2] registered [ 0.261183] acpiphp: Slot [5] registered [ 0.263129] acpiphp: Slot [6] registered [ 0.264095] acpiphp: Slot [7] registered [ 0.266206] acpiphp: Slot [8] registered [ 0.267113] acpiphp: Slot [9] registered [ 0.269106] acpiphp: Slot [10] registered [ 0.271107] acpiphp: Slot [3] registered [ 0.272098] acpiphp: Slot [4] registered [ 0.274098] acpiphp: Slot [11] registered [ 0.275119] acpiphp: Slot [12] registered [ 0.277094] acpiphp: Slot [13] registered [ 0.279092] acpiphp: Slot [14] registered [ 0.280090] acpiphp: Slot [15] registered [ 0.282069] acpiphp: Slot [16] registered [ 0.283093] acpiphp: Slot [17] registered [ 0.284063] acpiphp: Slot [18] registered [ 0.286116] acpiphp: Slot [19] registered [ 0.288099] acpiphp: Slot [20] registered [ 0.289083] acpiphp: Slot [21] registered [ 0.291111] acpiphp: Slot [22] registered [ 0.292086] acpiphp: Slot [23] registered [ 0.294103] acpiphp: Slot [24] registered [ 0.295101] acpiphp: Slot [25] registered [ 0.297104] acpiphp: Slot [26] registered [ 0.298137] acpiphp: Slot [27] registered [ 0.300084] acpiphp: Slot [28] registered [ 0.301082] acpiphp: Slot [29] registered [ 0.303085] acpiphp: Slot [30] registered [ 0.304084] acpiphp: Slot [31] registered [ 0.306081] PCI host bridge to bus 0000:00 [ 0.308032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.319022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321139] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323870] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327175] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336812] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341493] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.344016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.350608] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354862] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360827] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.382015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390061] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.429011] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.468014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.479800] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.491014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.522013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.530505] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.557986] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566788] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.574013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.612016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.621752] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.629013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.634014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669654] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672287] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676748] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.679401] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.683250] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.688048] iommu: Default domain type: Passthrough [ 0.689395] SCSI subsystem initialized [ 0.691168] ACPI: bus type USB registered [ 0.694116] usbcore: registered new interface driver usbfs [ 0.696078] usbcore: registered new interface driver hub [ 0.697079] usbcore: registered new device driver usb [ 0.699159] pps_core: LinuxPPS API ver. 1 registered [ 0.701010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.704054] PTP clock support registered [ 0.706056] EDAC MC: Ver: 3.0.0 [ 0.708151] PCI: Using ACPI for IRQ routing [ 0.710826] NetLabel: Initializing [ 0.712010] NetLabel: domain hash size = 128 [ 0.713007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.715077] NetLabel: unlabeled traffic allowed by default [ 0.717092] vgaarb: loaded [ 0.718273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.720014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727342] clocksource: Switched to clocksource kvm-clock [ 0.842072] VFS: Disk quotas dquot_6.6.0 [ 0.843832] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.847036] *** VALIDATE ramfs *** [ 0.848841] *** VALIDATE hugetlbfs *** [ 0.850639] pnp: PnP ACPI init [ 0.853345] pnp: PnP ACPI: found 6 devices [ 0.872872] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.878630] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.881230] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.883911] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.886835] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.889344] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.892739] NET: Registered protocol family 2 [ 0.895609] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.900903] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904550] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.909202] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.913134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.916369] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.919781] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923050] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927033] NET: Registered protocol family 1 [ 0.929733] RPC: Registered named UNIX socket transport module. [ 0.932030] RPC: Registered udp transport module. [ 0.933932] RPC: Registered tcp transport module. [ 0.935702] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.938028] NET: Registered protocol family 44 [ 0.940020] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.942349] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.944679] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.947097] PCI: CLS 0 bytes, default 64 [ 0.949276] Unpacking initramfs... [ 2.331982] debug: unmapping init [mem 0xffff8c3d3cc54000-0xffff8c3d3ffbffff] [ 2.336489] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.338466] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.340937] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.844723] Initialise system trusted keyrings [ 2.846324] Key type blacklist registered [ 2.848132] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.856304] zbud: loaded [ 2.859218] *** VALIDATE nfs *** [ 2.860224] *** VALIDATE nfs4 *** [ 2.862537] pstore: using deflate compression [ 2.865576] Platform Keyring initialized [ 2.975944] NET: Registered protocol family 38 [ 2.977957] Key type asymmetric registered [ 2.979494] Asymmetric key parser 'x509' registered [ 2.982104] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.985888] io scheduler mq-deadline registered [ 2.987918] io scheduler kyber registered [ 2.989647] io scheduler bfq registered [ 2.991651] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.994959] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.998057] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.001278] ACPI: Power Button [PWRF] [ 3.006755] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.014714] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.037476] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.046355] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.078913] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.107675] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.137375] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.142912] Non-volatile memory driver v1.3 [ 3.144926] Linux agpgart interface v0.103 [ 3.178310] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.180528] vda: detected capacity change from 0 to 76681216 [ 3.196363] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.199500] vdb: detected capacity change from 0 to 1073741824 [ 3.213056] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.216076] vdc: detected capacity change from 0 to 2621440000 [ 3.232692] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.236019] vdd: detected capacity change from 0 to 2621440000 [ 3.250408] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.253588] vde: detected capacity change from 0 to 4294967296 [ 3.268877] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272224] vdf: detected capacity change from 0 to 4294967296 [ 3.279469] libphy: Fixed MDIO Bus: probed [ 3.284884] usbcore: registered new interface driver usbserial_generic [ 3.287250] usbserial: USB Serial support registered for generic [ 3.289662] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.293819] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.295708] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.299128] mousedev: PS/2 mouse device common for all mice [ 3.302638] rtc_cmos 00:05: RTC can wake from S4 [ 3.306173] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.308389] rtc_cmos 00:05: registered as rtc0 [ 3.313199] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.315297] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317650] intel_pstate: CPU model not supported [ 3.322113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.326971] hid: raw HID events driver (C) Jiri Kosina [ 3.329189] usbcore: registered new interface driver usbhid [ 3.331329] usbhid: USB HID core driver [ 3.332721] drop_monitor: Initializing network drop monitor service [ 3.334882] Initializing XFRM netlink socket [ 3.336861] NET: Registered protocol family 10 [ 3.339879] Segment Routing with IPv6 [ 3.341483] NET: Registered protocol family 17 [ 3.343530] mpls_gso: MPLS GSO support [ 3.350108] RAS: Correctable Errors collector initialized. [ 3.352338] AVX version of gcm_enc/dec engaged. [ 3.354237] AES CTR mode by8 optimization enabled [ 3.434423] sched_clock: Marking stable (3434398503, 0)->(4333528810, -899130307) [ 3.438559] registered taskstats version 1 [ 3.440809] Loading compiled-in X.509 certificates [ 3.442696] zswap: loaded using pool lzo/zbud [ 3.466420] Key type big_key registered [ 3.480692] Key type encrypted registered [ 3.482245] ima: No TPM chip found, activating TPM-bypass! [ 3.485201] ima: Allocated hash algorithm: sha1 [ 3.487043] ima: No architecture policies found [ 3.488596] evm: Initialising EVM extended attributes: [ 3.490198] evm: security.selinux [ 3.491155] evm: security.ima [ 3.492153] evm: security.capability [ 3.493350] evm: HMAC attrs: 0x1 [ 3.495651] rtc_cmos 00:05: setting system clock to 2026-09-05 19:24:08 UTC (1788636248) [ 3.502084] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.504978] debug: unmapping init [mem 0xffffffff90982000-0xffffffff90c58fff] [ 3.511094] Write protecting the kernel read-only data: 28672k [ 3.514256] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 3.516720] debug: unmapping init [mem 0xffffffff8f914000-0xffffffff8f9fffff] [ 3.551666] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.560436] systemd[1]: Detected virtualization kvm. [ 3.562481] systemd[1]: Detected architecture x86-64. [ 3.564235] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.588901] systemd[1]: No hostname configured. [ 3.590868] systemd[1]: Set hostname to . [ 3.592879] random: systemd: uninitialized urandom read (16 bytes read) [ 3.595909] systemd[1]: Initializing machine ID from random generator. [ 3.648541] random: ln: uninitialized urandom read (6 bytes read) [ 3.750506] random: systemd: uninitialized urandom read (16 bytes read) [ 3.753505] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.758636] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.771410] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.448745] device-mapper: uevent: version 1.0.3 [ 4.451525] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.231912] random: fast init done [ 5.247660] virtio_net virtio0 ens2: renamed from eth0 [ 5.302461] scsi host0: ata_piix [ 5.356375] scsi host1: ata_piix [ 5.358508] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.361159] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.826463] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.998473] random: crng init done [ 10.000125] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.522101] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.707477] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.963393] SELinux: Disabled at runtime. [ 12.026379] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.036053] systemd[1]: Detected virtualization kvm. [ 12.038273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.537487] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.541832] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.547527] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.552989] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.556553] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.568144] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.577567] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.707924] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.062650] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.381291] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.471905] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.560270] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.586058] EDAC sbridge: Ver: 1.1.2 [ 15.321031] Key type dns_resolver registered [ 15.637884] NFS: Registering the id_resolver key type [ 15.640312] Key type id_resolver registered [ 15.641903] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-server login: [ 41.445693] libcfs: loading out-of-tree module taints kernel. [ 41.466951] Key type ._llcrypt registered [ 41.468593] Key type .llcrypt registered [ 41.523277] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 50.580373] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 51.352661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.362893] alg: No test for adler32 (adler32-zlib) [ 52.406424] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 52.778732] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 54.415213] Key type lgssc registered [ 56.387788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.890669] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.787991] hrtimer: interrupt took 2904431 ns [ 115.131566] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 127.765903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.781811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.164804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.217278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.330718] Lustre: lustre-MDT0000: new disk, initializing [ 129.462121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.498251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 134.214679] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.546622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.664856] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.694723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.697752] Lustre: Skipped 1 previous similar message [ 148.795481] Lustre: lustre-MDT0001: new disk, initializing [ 149.128144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.180776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.195149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 154.510123] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.225657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 169.952547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 170.244931] Lustre: lustre-OST0000: new disk, initializing [ 170.250822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 170.256629] Lustre: 8424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 170.325732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.694736] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.249948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 178.269425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 178.332337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.041843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.171683] Lustre: lustre-OST0001: new disk, initializing [ 190.175616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.185855] Lustre: 9497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.265232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.585853] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.213627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.219995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.284250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.465268] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.988907] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 228.416932] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing check_logdir /tmp/testlogs/ [ 234.098553] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing yml_node [ 238.144455] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 240.905466] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 243.603360] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 245.418091] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 15:28:10 EDT 2026 [ 263.396706] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 264.898146] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 281.064479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.076548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.085614] Lustre: Skipped 3 previous similar messages [ 281.110719] Lustre: Skipped 3 previous similar messages [ 286.176059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.188153] Lustre: Skipped 3 previous similar messages [ 286.783673] Lustre: server umount lustre-MDT0000 complete [ 289.986169] LustreError: 7433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636534 with bad export cookie 17923421344506188706 [ 289.986873] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.017235] LustreError: 7433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 291.300519] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.318250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 291.328849] Lustre: Skipped 1 previous similar message [ 294.308959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.326089] Lustre: Skipped 1 previous similar message [ 296.504433] Lustre: server umount lustre-MDT0001 complete [ 300.597308] Lustre: server umount lustre-OST0000 complete [ 304.260734] Lustre: server umount lustre-OST0001 complete [ 310.855464] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 320.322362] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 359.167658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.409864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.451633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.558284] Lustre: lustre-MDT0000: new disk, initializing [ 359.665130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.684326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 363.785907] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 374.210193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.320617] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.368724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.373768] Lustre: Skipped 1 previous similar message [ 374.474423] Lustre: lustre-MDT0001: new disk, initializing [ 374.574453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.641519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.662086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.002136] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 388.358511] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.434776] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 401.361918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 401.684464] Lustre: lustre-OST0000: new disk, initializing [ 401.690921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 401.703433] Lustre: 17022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 401.832071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.140705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 403.147072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 403.355427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 409.204450] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 418.729543] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 423.519958] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 423.799874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.585523] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 429.919767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 434.145874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.179715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.217760] Lustre: Skipped 1 previous similar message [ 434.250726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.670770] Lustre: server umount lustre-OST0000 complete [ 446.437971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.449067] LustreError: Skipped 1 previous similar message [ 446.455367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.468576] Lustre: Skipped 1 previous similar message [ 446.475982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.478598] Lustre: Skipped 3 previous similar messages [ 451.269748] Lustre: server umount lustre-MDT0000 complete [ 454.634254] LustreError: 15062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.664491] LustreError: 15062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 455.487701] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636700 with bad export cookie 17923421344506190232 [ 455.496456] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 455.501567] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 455.977927] Lustre: server umount lustre-MDT0001 complete [ 459.212383] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:31:44 (1788636704) [ 465.943352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.476863] LustreError: 19286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.545749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.931533] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 472.043862] LustreError: 19287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.130310] LustreError: 19286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.785738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.427623] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 491.377464] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.290463] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.683155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 510.447721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 519.475198] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 526.829071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.850589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.865394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.871680] Lustre: Skipped 2 previous similar messages [ 526.877382] Lustre: Skipped 2 previous similar messages [ 532.223589] Lustre: server umount lustre-OST0000 complete [ 536.545419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.557184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.367184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 550.576712] Lustre: server umount lustre-MDT0000 complete [ 552.416141] LustreError: 19286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.438106] LustreError: 19286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 554.464065] LustreError: 19267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636799 with bad export cookie 17923421344506191121 [ 554.470982] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.473772] LustreError: 19267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 554.889743] Lustre: server umount lustre-MDT0001 complete [ 566.366473] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 569.904769] Key type lgssc unregistered [ 570.237730] LNet: 22819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.255891] LNetError: 22819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.271750] LNet: Removed LNI 192.168.203.154@tcp [ 571.540286] Key type .llcrypt unregistered [ 571.544195] Key type ._llcrypt unregistered [ 595.705313] Key type ._llcrypt registered [ 595.707050] Key type .llcrypt registered [ 595.832784] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 597.117598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 597.173550] alg: No test for adler32 (adler32-zlib) [ 598.391708] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 598.648075] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 600.312081] Key type lgssc registered [ 601.312306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.225787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 609.255908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.772370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.903096] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 620.006192] LustreError: 24227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.044280] LustreError: 24227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 624.600829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.985137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 629.441954] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 636.560707] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 641.573968] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 647.278357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 647.646837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.968654] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 655.877312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 661.693302] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 680.028528] Lustre: server umount lustre-OST0000 complete [ 686.564920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.570652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 686.582580] Lustre: Skipped 1 previous similar message [ 686.589956] Lustre: Skipped 1 previous similar message [ 691.686447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 691.703248] Lustre: Skipped 1 previous similar message [ 694.838582] Lustre: server umount lustre-MDT0000 complete [ 696.799793] Lustre: 23539:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636925/real 1788636925] req@00000000f34534df x1875521666315392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788636941 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 696.809868] LustreError: 24221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.827517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.861245] LustreError: 24221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 697.828291] Lustre: 23538:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636926/real 1788636926] req@0000000078e14028 x1875521666316160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788636942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 698.336316] LustreError: 27250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636943 with bad export cookie 16246510991196619538 [ 698.340448] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.345278] LustreError: 27250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 698.700435] Lustre: server umount lustre-MDT0001 complete [ 708.486376] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 711.488190] Key type lgssc unregistered [ 711.825792] LNet: 28200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.837351] LNetError: 28200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.852193] LNet: Removed LNI 192.168.203.154@tcp [ 712.783611] Key type .llcrypt unregistered [ 712.788112] Key type ._llcrypt unregistered [ 737.224553] Key type ._llcrypt registered [ 737.229848] Key type .llcrypt registered [ 737.345342] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 738.394978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.514197] alg: No test for adler32 (adler32-zlib) [ 739.634370] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 739.908273] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 741.607141] Key type lgssc registered [ 742.787037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 750.521022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.537066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.046728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.074753] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 762.340886] LustreError: 29611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.362164] LustreError: 29611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 764.225145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.639622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 769.525322] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 776.856279] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 781.807201] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 794.455724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 794.830302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.937957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 800.038120] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 806.182614] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 818.657176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 818.674081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.694036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.681673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.689031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.689662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 822.732462] Lustre: server umount lustre-OST0000 complete [ 826.336071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 826.342847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.360219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.364257] Lustre: Skipped 1 previous similar message [ 829.920581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 829.922780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.041670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.048229] Lustre: Skipped 2 previous similar messages [ 840.159596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 840.311773] Lustre: server umount lustre-MDT0000 complete [ 843.279495] LustreError: 29591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637088 with bad export cookie 7212439160598096548 [ 843.286862] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 843.303395] LustreError: 29591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 843.586464] Lustre: server umount lustre-MDT0001 complete [ 852.683918] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 855.173743] Key type lgssc unregistered [ 855.396784] LNet: 33487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.405365] LNetError: 33487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.418849] LNet: Removed LNI 192.168.203.154@tcp [ 856.128183] Key type .llcrypt unregistered [ 856.136536] Key type ._llcrypt unregistered [ 876.828659] Key type ._llcrypt registered [ 876.834978] Key type .llcrypt registered [ 876.968026] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 877.746224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.772892] alg: No test for adler32 (adler32-zlib) [ 878.774198] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 878.976476] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 880.655326] Key type lgssc registered [ 881.510252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.584399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.619831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.020080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.792966] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 898.529743] LustreError: 34876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.558241] LustreError: 34876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 900.322859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.619773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.316306] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 911.309483] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 916.552024] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.071494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.319688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.684922] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 933.548404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 935.189797] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 944.100960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 944.102123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.108543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.121839] Lustre: Skipped 1 previous similar message [ 948.099953] Lustre: server umount lustre-OST0000 complete [ 951.777552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 951.785260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.812594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.820806] Lustre: Skipped 1 previous similar message [ 954.337781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.338800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.668498] Lustre: server umount lustre-MDT0000 complete [ 959.457511] LustreError: 34875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.465377] LustreError: 34875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 960.128158] LustreError: 35612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637205 with bad export cookie 16346599167811771049 [ 960.130486] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.139791] LustreError: 35612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 960.334624] Lustre: server umount lustre-MDT0001 complete [ 970.240934] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 973.513320] Key type lgssc unregistered [ 974.017665] LNet: 38409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.036501] LNetError: 38409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.064081] LNet: Removed LNI 192.168.203.154@tcp [ 975.288334] Key type .llcrypt unregistered [ 975.291394] Key type ._llcrypt unregistered [ 1000.964649] Key type ._llcrypt registered [ 1000.967320] Key type .llcrypt registered [ 1001.079510] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1001.744249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.773313] alg: No test for adler32 (adler32-zlib) [ 1002.725715] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1002.997719] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1004.769761] Key type lgssc registered [ 1005.909265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.422461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.479427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.097362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.128524] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1026.535828] LustreError: 39820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.552056] LustreError: 39820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1029.727720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.100925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1034.770212] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1042.961729] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1049.241220] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1055.394684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.652758] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1055.761394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.858626] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1061.890582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1069.352386] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1076.194919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.203492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.218371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.219202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.220748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.233510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.265567] Lustre: Skipped 2 previous similar messages [ 1081.454663] Lustre: server umount lustre-OST0000 complete [ 1101.792738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1101.804603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.817802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.861141] Lustre: server umount lustre-MDT0000 complete [ 1107.944815] LustreError: 39816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.965267] LustreError: 39816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1110.079988] LustreError: 42399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637355 with bad export cookie 18352670157155441210 [ 1110.089408] LustreError: 42399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1110.090193] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1116.488982] Lustre: server umount lustre-MDT0001 complete [ 1126.787085] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1129.298318] Key type lgssc unregistered [ 1129.517274] LNet: 43526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.524556] LNetError: 43526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.598735] LNet: Removed LNI 192.168.203.154@tcp [ 1131.349383] Key type .llcrypt unregistered [ 1131.352718] Key type ._llcrypt unregistered [ 1149.229699] Key type ._llcrypt registered [ 1149.232366] Key type .llcrypt registered [ 1149.295709] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1149.777315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.784924] alg: No test for adler32 (adler32-zlib) [ 1150.715736] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1150.885231] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1152.511169] Key type lgssc registered [ 1153.367635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.569297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.587316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.897164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.372293] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1170.401901] LustreError: 44913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.421388] LustreError: 44913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1171.643802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.002630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1175.926332] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1182.116524] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.243735] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1193.500844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.939524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.803478] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1205.241173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1206.200519] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1212.896221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1212.902235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.910082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.462396] Lustre: server umount lustre-MDT0000 complete [ 1215.459722] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.461894] LustreError: 44913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.475141] Lustre: Skipped 1 previous similar message [ 1215.485958] LustreError: 44913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1216.865651] LustreError: 44893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637461 with bad export cookie 3826779452000365170 [ 1216.878038] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.153599] LustreError: 44910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.173784] LustreError: 44910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1218.182158] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 1218.197888] Lustre: Skipped 1 previous similar message [ 1220.577447] LustreError: 44908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.579358] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.599283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1223.273246] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 1223.288945] Lustre: Skipped 1 previous similar message [ 1226.721471] LustreError: 44909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.737725] LustreError: 44909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1228.398062] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 1228.408778] Lustre: Skipped 1 previous similar message [ 1231.327293] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1231.543275] Lustre: server umount lustre-MDT0001 complete [ 1246.304416] Lustre: server umount lustre-OST0000 complete [ 1261.214929] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1263.766341] Key type lgssc unregistered [ 1264.098326] LNet: 48768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.105309] LNetError: 48768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.123815] LNet: Removed LNI 192.168.203.154@tcp [ 1265.040966] Key type .llcrypt unregistered [ 1265.043459] Key type ._llcrypt unregistered [ 1286.813810] Key type ._llcrypt registered [ 1286.816453] Key type .llcrypt registered [ 1286.905669] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1287.952981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.983884] alg: No test for adler32 (adler32-zlib) [ 1289.044693] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1289.278521] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1290.983165] Key type lgssc registered [ 1291.931373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.201379] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.229650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.760323] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1318.879509] LustreError: 50130:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 1345.567191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.529095] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1358.208920] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1525.852328] Lustre: server umount lustre-OST0000 complete [ 1541.096347] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1543.437949] Key type lgssc unregistered [ 1543.699941] LNet: 51805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.716697] LNetError: 51805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.742852] LNet: Removed LNI 192.168.203.154@tcp [ 1544.581319] Key type .llcrypt unregistered [ 1544.583922] Key type ._llcrypt unregistered [ 1566.072631] Key type ._llcrypt registered [ 1566.074762] Key type .llcrypt registered [ 1566.163709] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1566.956384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1567.046188] alg: No test for adler32 (adler32-zlib) [ 1568.020307] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1568.278086] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1570.000130] Key type lgssc registered [ 1571.472471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1582.182560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1582.224484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.697956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.135873] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1593.834141] LustreError: 53207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.862611] LustreError: 53207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1598.187273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.614817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.655894] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1611.853987] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1618.704463] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1625.970236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.480877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.540797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1633.271051] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1641.358787] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1644.003244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.017422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.032369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1649.121193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1649.122558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1649.136546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.243492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1654.260775] Lustre: Skipped 1 previous similar message [ 1658.335123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1658.463442] Lustre: server umount lustre-OST0000 complete [ 1664.480634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.481591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.500190] Lustre: Skipped 1 previous similar message [ 1664.511597] Lustre: Skipped 2 previous similar messages [ 1668.298349] Lustre: server umount lustre-MDT0000 complete [ 1669.607122] LustreError: 53935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.642289] LustreError: 53935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1672.387620] LustreError: 53188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637917 with bad export cookie 1197766443684905237 [ 1672.391601] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.407961] LustreError: 53188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1672.941571] Lustre: server umount lustre-MDT0001 complete [ 1683.394769] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1686.506395] Key type lgssc unregistered [ 1686.783583] LNet: 56738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.788585] LNetError: 56738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.809705] LNet: Removed LNI 192.168.203.154@tcp [ 1687.858263] Key type .llcrypt unregistered [ 1687.865117] Key type ._llcrypt unregistered [ 1716.054593] Key type ._llcrypt registered [ 1716.059659] Key type .llcrypt registered [ 1716.187088] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1717.704898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.744764] alg: No test for adler32 (adler32-zlib) [ 1718.964039] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1719.428527] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1721.231157] Key type lgssc registered [ 1722.812575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1731.645189] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1731.682432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.232107] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1752.351312] LustreError: 58125:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 1778.975465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.845566] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1793.201824] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1802.121988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.838984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.923200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1806.853426] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1807.845108] LustreError: 58150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.876389] LustreError: 58150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1812.964249] LustreError: 58151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.991381] LustreError: 59161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.033498] LustreError: 59161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1815.164972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.490116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.451475] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1828.616924] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1835.902153] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1838.560868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1838.571187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.594210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.120633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.121699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.138054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.160684] Lustre: Skipped 1 previous similar message [ 1846.247594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.267680] Lustre: Skipped 1 previous similar message [ 1851.367187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1851.382818] Lustre: Skipped 1 previous similar message [ 1852.383731] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1852.600988] Lustre: server umount lustre-OST0000 complete [ 1876.965468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1876.975558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.979318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.994825] Lustre: Skipped 1 previous similar message [ 1877.007449] Lustre: Skipped 1 previous similar message [ 1879.920425] Lustre: server umount lustre-MDT0000 complete [ 1881.130207] LustreError: 59167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.145028] LustreError: 59167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1883.031747] LustreError: 58133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638128 with bad export cookie 17094750585915091713 [ 1883.036541] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1883.039785] LustreError: 58133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1889.669318] Lustre: server umount lustre-MDT0001 complete [ 1901.710643] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1904.779884] Key type lgssc unregistered [ 1905.130386] LNet: 61868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.137603] LNetError: 61868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.168650] LNet: Removed LNI 192.168.203.154@tcp [ 1905.856145] Key type .llcrypt unregistered [ 1905.859923] Key type ._llcrypt unregistered [ 1929.761262] Key type ._llcrypt registered [ 1929.769362] Key type .llcrypt registered [ 1929.910643] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1931.267795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.326401] alg: No test for adler32 (adler32-zlib) [ 1932.386736] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1932.713444] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1934.407180] Key type lgssc registered [ 1935.784977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.700894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1946.734267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.635794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.855170] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1958.882501] LustreError: 63283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.901414] LustreError: 63283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1963.850278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.076402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.946259] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1976.192765] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.512749] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1987.885031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.242451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.424841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1994.261039] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2003.753754] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.420566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2024.424456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.436634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2024.483082] Lustre: Skipped 1 previous similar message [ 2029.537468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2029.544835] Lustre: Skipped 2 previous similar messages [ 2034.659483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.683203] Lustre: Skipped 1 previous similar message [ 2037.218271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2037.473942] Lustre: server umount lustre-OST0000 complete [ 2044.897397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.926397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.934377] Lustre: Skipped 1 previous similar message [ 2047.422059] Lustre: server umount lustre-MDT0000 complete [ 2050.017367] LustreError: 63278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.027914] LustreError: 63278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2051.200031] LustreError: 63263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638296 with bad export cookie 9596881685145576078 [ 2051.202385] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.208257] LustreError: 63263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2051.758107] Lustre: server umount lustre-MDT0001 complete [ 2063.285082] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2066.148347] Key type lgssc unregistered [ 2066.507324] LNet: 66841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.521508] LNetError: 66841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.575310] LNet: Removed LNI 192.168.203.154@tcp [ 2067.542183] Key type .llcrypt unregistered [ 2067.544677] Key type ._llcrypt unregistered [ 2097.807459] Key type ._llcrypt registered [ 2097.815673] Key type .llcrypt registered [ 2097.979818] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2098.900717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.046572] alg: No test for adler32 (adler32-zlib) [ 2100.221809] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2100.528556] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2102.225315] Key type lgssc registered [ 2103.267743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2112.970088] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 15:59:17 (1788638357) [ 2119.211415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2119.223853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.679713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.772946] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2130.922423] LustreError: 68545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.933733] LustreError: 68545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2132.274086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.659204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.975189] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2143.682766] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2149.778032] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2157.815959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.233251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.308989] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2165.426426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2173.938350] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2183.166607] Lustre: Failing over lustre-MDT0000 [ 2183.977760] Lustre: server umount lustre-MDT0000 complete [ 2184.163714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2184.185634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.219296] LustreError: 68545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.888204] LustreError: 68541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.258241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.260892] LustreError: 69272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.345623] LustreError: 69272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2193.220581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.381427] LustreError: 68546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.421310] LustreError: 68546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2193.524367] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2193.850907] Lustre: *** cfs_fail_loc=135, val=0*** [ 2194.050806] LustreError: 71632:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2194.072834] LustreError: 71632:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.154@tcp: cfg command failed: rc = -2 [ 2194.090885] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2194.090885] [ 2194.101951] LustreError: MGC192.168.203.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2194.123403] LustreError: 71620:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2194.137711] LustreError: 71620:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2194.157074] LustreError: 71620:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2194.172914] LustreError: 71620:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2194.213411] Lustre: server umount lustre-MDT0000 complete [ 2194.217376] LustreError: 71620:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2201.554668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.646683] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.654868] LustreError: 68538:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2201.943029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.008226] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2202.267147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2206.294402] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2207.210127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2207.233634] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2207.256851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2217.444924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.448888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.454984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2217.472090] Lustre: Skipped 2 previous similar messages [ 2220.858793] Lustre: server umount lustre-OST0000 complete [ 2227.680988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.699035] Lustre: Skipped 1 previous similar message [ 2227.700539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.712933] Lustre: Skipped 1 previous similar message [ 2230.732093] Lustre: server umount lustre-MDT0000 complete [ 2232.801195] LustreError: 68545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.826259] LustreError: 68545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2234.450799] LustreError: 68527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638479 with bad export cookie 8563848657257990541 [ 2234.464343] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.472261] LustreError: 68527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2235.004545] Lustre: server umount lustre-MDT0001 complete [ 2247.154293] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2251.268789] Key type lgssc unregistered [ 2251.653195] LNet: 73422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.665496] LNetError: 73422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.689429] LNet: Removed LNI 192.168.203.154@tcp [ 2253.033518] Key type .llcrypt unregistered [ 2253.042232] Key type ._llcrypt unregistered [ 2277.043606] Key type ._llcrypt registered [ 2277.046689] Key type .llcrypt registered [ 2277.129643] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2277.978819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.056054] alg: No test for adler32 (adler32-zlib) [ 2279.112051] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2279.443088] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2281.079155] Key type lgssc registered [ 2281.942830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.439829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.465297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.892769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.014470] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2300.387581] LustreError: 74832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.401437] LustreError: 74832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2304.095241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.529733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.309354] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2316.858328] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2322.433583] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2329.160256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.422932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.966902] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2335.611314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2343.508669] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2349.676862] Lustre: Failing over lustre-MDT0000 [ 2350.307418] Lustre: server umount lustre-MDT0000 complete [ 2350.568926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2350.577893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.598930] LustreError: 74832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.084602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.102565] Lustre: Skipped 1 previous similar message [ 2355.825518] LustreError: 74825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.846153] LustreError: 74825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2359.716539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.812303] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.918622] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2360.067867] LustreError: 77920:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2360.074342] LustreError: 77920:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.154@tcp: cfg command failed: rc = -9 [ 2360.079585] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2360.079585] [ 2360.096676] LustreError: MGC192.168.203.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2360.117870] LustreError: 77908:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2360.127145] LustreError: 77908:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2360.132184] LustreError: 77908:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2360.138669] LustreError: 77908:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2360.156899] Lustre: server umount lustre-MDT0000 complete [ 2360.161155] LustreError: 77908:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2360.941054] LustreError: 74825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.969433] LustreError: 74825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2366.073174] LustreError: 74828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.102694] LustreError: 74828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2366.939601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.122312] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.524888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.598579] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2371.170606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2372.606635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2372.643988] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2372.687144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2372.693932] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2381.805848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2381.816091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.829555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.824566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.846737] Lustre: Skipped 1 previous similar message [ 2386.941964] Lustre: server umount lustre-OST0000 complete [ 2393.060081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.072759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.075041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.118748] Lustre: Skipped 2 previous similar messages [ 2398.178289] LustreError: 74828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.194608] LustreError: 74828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2399.052571] Lustre: server umount lustre-MDT0000 complete [ 2403.109742] LustreError: 74811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638648 with bad export cookie 3226435298187389080 [ 2403.116310] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.119471] LustreError: 74811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2403.642154] Lustre: server umount lustre-MDT0001 complete [ 2415.799214] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2418.839550] Key type lgssc unregistered [ 2419.241847] LNet: 79701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.256023] LNetError: 79701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.279315] LNet: Removed LNI 192.168.203.154@tcp [ 2420.227403] Key type .llcrypt unregistered [ 2420.232235] Key type ._llcrypt unregistered [ 2434.949063] Key type ._llcrypt registered [ 2434.952352] Key type .llcrypt registered [ 2435.088681] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2435.433428] Key type .llcrypt unregistered [ 2435.437340] Key type ._llcrypt unregistered [ 2461.402396] Key type ._llcrypt registered [ 2461.405463] Key type .llcrypt registered [ 2461.532454] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2462.864312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.887110] alg: No test for adler32 (adler32-zlib) [ 2464.036997] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2464.250247] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2465.943743] Key type lgssc registered [ 2467.600365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.165430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.184561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.632938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.649278] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2486.755919] LustreError: 81600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.782108] LustreError: 81600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2487.876020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.221561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.305899] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2499.326781] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2505.923963] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2512.815508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.192306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.946548] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2521.409498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2527.768845] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2540.000556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2540.014685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.026499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.049883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2542.059084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.059222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.086093] Lustre: Skipped 1 previous similar message [ 2544.617516] Lustre: server umount lustre-OST0000 complete [ 2549.728720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.743180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.759776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.288418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.293911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.816548] Lustre: server umount lustre-MDT0000 complete [ 2557.415231] LustreError: 81600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.657480] LustreError: 81581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638803 with bad export cookie 11316225482787158747 [ 2558.669450] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.681249] LustreError: 81581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2559.183812] Lustre: server umount lustre-MDT0001 complete [ 2569.294657] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2572.045302] Key type lgssc unregistered [ 2572.346056] LNet: 85133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2572.351823] LNetError: 85133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.421842] LNet: Removed LNI 192.168.203.154@tcp [ 2574.359495] Key type .llcrypt unregistered [ 2574.362201] Key type ._llcrypt unregistered [ 2601.018691] Key type ._llcrypt registered [ 2601.020697] Key type .llcrypt registered [ 2601.162965] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2602.091491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2602.138282] alg: No test for adler32 (adler32-zlib) [ 2603.414087] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2603.659840] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2605.359165] Key type lgssc registered [ 2606.696536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2615.217153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2615.269486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.851383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.051209] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2626.021127] LustreError: 86541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.075890] LustreError: 86541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2631.137636] LustreError: 86542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.569639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.959339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2636.716259] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2645.543220] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.615631] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.718490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.148934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.209600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2668.481960] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2677.061549] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2686.945243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.946246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2686.956605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2686.962427] Lustre: Skipped 1 previous similar message [ 2689.221780] Lustre: server umount lustre-OST0000 complete [ 2697.188602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.190711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.235353] Lustre: Skipped 2 previous similar messages [ 2701.146173] Lustre: server umount lustre-MDT0000 complete [ 2702.304631] LustreError: 89031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.578459] LustreError: 86521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638950 with bad export cookie 7133102571335389487 [ 2705.580491] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.605275] LustreError: 86521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2705.981882] Lustre: server umount lustre-MDT0001 complete [ 2720.496858] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2724.009719] Key type lgssc unregistered [ 2724.273971] LNet: 90074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.286115] LNetError: 90074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.309139] LNet: Removed LNI 192.168.203.154@tcp [ 2725.344384] Key type .llcrypt unregistered [ 2725.346464] Key type ._llcrypt unregistered [ 2751.108533] Key type ._llcrypt registered [ 2751.115050] Key type .llcrypt registered [ 2751.222508] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2752.388704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2752.450092] alg: No test for adler32 (adler32-zlib) [ 2753.618161] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2753.965464] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2755.760330] Key type lgssc registered [ 2757.121474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.798447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.832474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.353101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.245684] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2777.580583] LustreError: 91486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.615397] LustreError: 91486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2781.774640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.239233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.000887] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2795.406820] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2802.433142] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2810.740132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.132970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.154368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2817.953576] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2827.322573] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2841.065186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2841.071705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.098337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.106982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2843.110409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.114975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.176908] Lustre: Skipped 1 previous similar message [ 2848.225318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.232098] Lustre: Skipped 1 previous similar message [ 2853.379829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2853.397255] Lustre: Skipped 1 previous similar message [ 2854.879126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2855.061018] Lustre: server umount lustre-OST0000 complete [ 2858.978280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2858.987343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.010408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.589604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.706795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.721369] Lustre: Skipped 3 previous similar messages [ 2873.311144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2873.552123] Lustre: server umount lustre-MDT0000 complete [ 2873.830168] LustreError: 93842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.866469] LustreError: 93842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2877.575531] LustreError: 91465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639122 with bad export cookie 9751738636874071447 [ 2877.581686] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.598448] LustreError: 91465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2878.134413] Lustre: server umount lustre-MDT0001 complete [ 2893.333889] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2896.501253] Key type lgssc unregistered [ 2896.940166] LNet: 95041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2896.946590] LNetError: 95041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2896.979964] LNet: Removed LNI 192.168.203.154@tcp [ 2897.879914] Key type .llcrypt unregistered [ 2897.881981] Key type ._llcrypt unregistered [ 2920.818847] Key type ._llcrypt registered [ 2920.820902] Key type .llcrypt registered [ 2920.962656] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2922.015924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2922.175354] alg: No test for adler32 (adler32-zlib) [ 2923.273254] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2923.525452] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2925.231173] Key type lgssc registered [ 2926.303681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2934.510711] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2934.528580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2950.047470] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2955.167347] LustreError: 96416:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 2981.855902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.210915] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2998.046100] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3004.137835] Lustre: server umount lustre-OST0000 complete [ 3013.442533] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:14:17 (1788639257) [ 3020.013313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.460655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.288336] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3025.893803] LustreError: 98055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.913266] LustreError: 98055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3031.019584] LustreError: 98056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.366871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.841955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3040.968295] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3050.309030] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3056.990713] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3065.451937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.911216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.938358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3072.942521] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3081.705679] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3095.524105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.529287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.545824] Lustre: Skipped 1 previous similar message [ 3095.562381] Lustre: Skipped 1 previous similar message [ 3099.575810] Lustre: server umount lustre-OST0000 complete [ 3105.760957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.764823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.774133] Lustre: Skipped 1 previous similar message [ 3110.157852] Lustre: server umount lustre-MDT0000 complete [ 3110.882214] LustreError: 99052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.905155] LustreError: 99052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3114.070647] LustreError: 98036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639359 with bad export cookie 743686492804945600 [ 3114.076124] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.096039] LustreError: 98036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3114.525263] Lustre: server umount lustre-MDT0001 complete [ 3126.646029] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3130.187883] Key type lgssc unregistered [ 3130.562518] LNet: 101750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3130.571874] LNetError: 101750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.592310] LNet: Removed LNI 192.168.203.154@tcp [ 3131.566617] Key type .llcrypt unregistered [ 3131.576300] Key type ._llcrypt unregistered [ 3181.375316] Key type ._llcrypt registered [ 3181.377624] Key type .llcrypt registered [ 3181.493246] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3182.713349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3182.759714] alg: No test for adler32 (adler32-zlib) [ 3183.790938] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3184.105667] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3185.871250] Key type lgssc registered [ 3187.277480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3195.853526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3195.883839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.504500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.611795] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3207.650633] LustreError: 103932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.671697] LustreError: 103932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3211.550965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.993900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3217.230977] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3226.303974] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3232.402828] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3239.310160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.780090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.866510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3246.380445] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3254.149665] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3263.471654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3263.478473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.493935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.528112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.530614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.543908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.584372] Lustre: Skipped 1 previous similar message [ 3269.596593] Lustre: server umount lustre-OST0000 complete [ 3273.699365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3273.707925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.723852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.770749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.775348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.183505] Lustre: server umount lustre-MDT0000 complete [ 3281.895159] LustreError: 103932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.922153] LustreError: 103932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3284.369310] LustreError: 105777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639529 with bad export cookie 14845890409553513141 [ 3284.399116] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.057276] Lustre: server umount lustre-MDT0001 complete [ 3298.148885] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3301.225543] Key type lgssc unregistered [ 3301.517662] LNet: 107465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.556345] LNetError: 107465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.583313] LNet: Removed LNI 192.168.203.154@tcp [ 3303.017175] Key type .llcrypt unregistered [ 3303.022467] Key type ._llcrypt unregistered [ 3324.030979] Key type ._llcrypt registered [ 3324.033390] Key type .llcrypt registered [ 3324.192296] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3325.209327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3325.401653] alg: No test for adler32 (adler32-zlib) [ 3326.557260] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3326.854663] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3328.536197] Key type lgssc registered [ 3329.853357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3337.323685] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3337.349788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.864236] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3357.985125] LustreError: 108705:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 3384.607182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.879255] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3400.190672] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3407.330636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.696928] LustreError: MGC192.168.203.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3407.706073] LustreError: 109731:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3407.710593] LustreError: 109731:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3407.718075] LustreError: 109731:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3407.776366] Lustre: server umount lustre-MDT0000 complete [ 3407.779247] LustreError: 109731:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3426.411559] Lustre: server umount lustre-OST0000 complete [ 3437.353056] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3450.390075] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3490.694198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.968397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3491.003342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3491.091377] Lustre: lustre-MDT0000: new disk, initializing [ 3491.177774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.217985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3496.247902] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3508.690040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.896867] Lustre: 112526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3508.988741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3508.992881] Lustre: Skipped 1 previous similar message [ 3509.129167] Lustre: lustre-MDT0001: new disk, initializing [ 3509.235727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.289814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3509.314403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3514.266230] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3524.471210] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3530.939208] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3538.809709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.341704] Lustre: lustre-OST0000: new disk, initializing [ 3539.346593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3539.359583] Lustre: 114494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3539.493952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.146566] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3549.826138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3549.841762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3549.957258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3557.692307] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3562.608677] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3562.809780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3567.952458] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3568.139814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3570.658626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.662899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.668119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.699186] Lustre: Skipped 1 previous similar message [ 3575.788333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.802510] Lustre: Skipped 2 previous similar messages [ 3576.348517] Lustre: server umount lustre-OST0000 complete [ 3580.897044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.904215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3580.909745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.019587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.029575] Lustre: Skipped 2 previous similar messages [ 3586.862685] Lustre: server umount lustre-MDT0000 complete [ 3590.919292] LustreError: 112519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639835 with bad export cookie 4896135001936422542 [ 3590.924790] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3590.929463] LustreError: 112519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3591.329865] Lustre: server umount lustre-MDT0001 complete [ 3600.861824] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 16:24:05 (1788639845) [ 3604.524334] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3621.466343] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3632.268232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3673.827767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.322779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3674.377688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3674.572971] Lustre: lustre-MDT0000: new disk, initializing [ 3674.909192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.946200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3680.791538] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3692.296546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.475763] Lustre: 119396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3692.549770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3692.554764] Lustre: Skipped 1 previous similar message [ 3692.662705] Lustre: lustre-MDT0001: new disk, initializing [ 3692.777398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3692.834640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3692.858420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3697.361891] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3707.528513] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3713.878434] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3721.571463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.800086] Lustre: lustre-OST0000: new disk, initializing [ 3721.805892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3721.816669] Lustre: 121366:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3721.895693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.856864] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3728.986111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3728.998685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3729.131249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3738.941962] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3743.967697] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3744.196534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3749.285974] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3749.557438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3754.978862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.991430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.992944] LustreError: Skipped 1 previous similar message [ 3754.993529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.027433] Lustre: Skipped 2 previous similar messages [ 3758.062720] Lustre: server umount lustre-OST0000 complete [ 3764.704658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3764.714579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.733991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.736899] Lustre: Skipped 1 previous similar message [ 3768.697313] Lustre: server umount lustre-MDT0000 complete [ 3772.434490] LustreError: 120190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640017 with bad export cookie 4896135001936423431 [ 3772.435455] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.454319] LustreError: 120190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3772.970956] Lustre: server umount lustre-MDT0001 complete [ 3782.727385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.288611] LustreError: 123582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.311633] LustreError: 123582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3783.361806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.782022] LustreError: 123583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.873241] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3793.890946] LustreError: 123582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.975552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.944581] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3810.594580] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3816.673826] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.395574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.182311] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3837.353556] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3847.434500] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3853.800937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3853.801417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.810758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.823794] Lustre: Skipped 2 previous similar messages [ 3853.836636] Lustre: Skipped 3 previous similar messages [ 3857.886365] Lustre: server umount lustre-OST0000 complete [ 3864.041416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.319353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3876.543900] Lustre: server umount lustre-MDT0000 complete [ 3878.375213] LustreError: 124312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.391567] LustreError: 124312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3880.449875] LustreError: 123564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640125 with bad export cookie 4896135001936424313 [ 3880.458552] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.467021] LustreError: 123564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.811523] Lustre: server umount lustre-MDT0001 complete [ 3891.934138] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3894.497547] Key type lgssc unregistered [ 3894.860944] LNet: 127213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3894.872227] LNetError: 127213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3895.913132] LNet: Removed LNI 192.168.203.154@tcp [ 3896.938592] Key type .llcrypt unregistered [ 3896.940258] Key type ._llcrypt unregistered [ 3915.944954] Key type ._llcrypt registered [ 3915.949949] Key type .llcrypt registered [ 3916.058725] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3932.734150] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3934.082881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3934.159730] alg: No test for adler32 (adler32-zlib) [ 3935.301653] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3935.647766] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3937.351252] Key type lgssc registered [ 3938.580841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3972.282329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3972.313709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.499669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3973.524646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3973.593625] Lustre: lustre-MDT0000: new disk, initializing [ 3973.692367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.721557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3977.753085] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3987.428847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.548885] Lustre: 130373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3987.625166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3987.630263] Lustre: Skipped 1 previous similar message [ 3987.702692] Lustre: lustre-MDT0001: new disk, initializing [ 3987.805806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3987.855122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3987.875217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3991.266928] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4000.889251] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4007.864548] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4015.759990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.043107] Lustre: lustre-OST0000: new disk, initializing [ 4016.050188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4016.062545] Lustre: 132342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4016.238647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.824271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4021.835959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4021.938640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4024.073951] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4033.799747] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4038.774425] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4038.999985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4043.614256] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4043.869420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4047.854269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4047.859116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.872306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4047.888150] Lustre: Skipped 1 previous similar message [ 4047.914587] Lustre: Skipped 1 previous similar message [ 4052.153222] Lustre: server umount lustre-OST0000 complete [ 4058.082557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.097590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4058.103241] Lustre: Skipped 1 previous similar message [ 4061.966724] Lustre: server umount lustre-MDT0000 complete [ 4065.739790] LustreError: 130366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640310 with bad export cookie 6881907291472816783 [ 4065.758684] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.762371] LustreError: 130366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4066.150823] Lustre: server umount lustre-MDT0001 complete [ 4074.736633] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 16:31:59 (1788640319) [ 4081.294435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.767778] LustreError: 134756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.792517] LustreError: 134756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4081.868478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.598304] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4087.271992] LustreError: 134757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.372403] LustreError: 134756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.338684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.802169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4100.486062] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4109.528326] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4117.476412] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4121.186466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.191434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.197218] Lustre: Skipped 2 previous similar messages [ 4121.218159] Lustre: Skipped 1 previous similar message [ 4126.505915] Lustre: server umount lustre-MDT0000 complete [ 4126.697681] LustreError: 134751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.708956] LustreError: 134751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4130.723802] LustreError: 134736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640375 with bad export cookie 6881907291472817665 [ 4130.728481] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4130.733077] LustreError: 134736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4131.188625] Lustre: server umount lustre-MDT0001 complete [ 4142.420085] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 16:33:07 (1788640387) [ 4151.549686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.137534] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4172.260207] LustreError: 137174:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 4197.983818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.143697] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4213.100495] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4214.979399] Lustre: server umount lustre-OST0000 complete [ 4225.424907] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 16:34:29 (1788640469) [ 4232.759750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.353904] LustreError: 138620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.440305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.083836] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4243.952935] LustreError: 138621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4243.967084] LustreError: 138621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4247.577814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.606371] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4260.058744] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4265.771156] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4272.097638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4272.390132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.397607] Lustre: Skipped 1 previous similar message [ 4277.924785] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4285.577612] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.242949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.246958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4299.250519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4299.250529] LustreError: Skipped 1 previous similar message [ 4299.272279] Lustre: Skipped 2 previous similar messages [ 4299.280582] Lustre: Skipped 1 previous similar message [ 4301.356569] Lustre: server umount lustre-MDT0000 complete [ 4304.359226] LustreError: 138621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.380986] LustreError: 138621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4305.231092] LustreError: 140464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640550 with bad export cookie 6881907291472818330 [ 4305.235839] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.245210] LustreError: 140464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4305.880530] Lustre: server umount lustre-MDT0001 complete [ 4319.786938] Lustre: server umount lustre-OST0000 complete [ 4328.066978] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 16:36:12 (1788640572) [ 4333.913583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4338.898701] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4339.681685] LustreError: 142273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.707736] LustreError: 142273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4347.937610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.277057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4348.287510] Lustre: Skipped 1 previous similar message [ 4352.972252] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4360.379633] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4366.365259] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4373.543947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4379.069950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4380.081135] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4388.636858] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4393.837823] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4394.057948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4399.615976] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4399.920234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4404.711859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4404.728565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.744887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4404.748939] Lustre: Skipped 1 previous similar message [ 4404.763054] Lustre: Skipped 2 previous similar messages [ 4408.427495] Lustre: server umount lustre-OST0000 complete [ 4414.947149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4414.952650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4414.965163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4414.971721] Lustre: Skipped 1 previous similar message [ 4418.585403] Lustre: server umount lustre-MDT0000 complete [ 4420.074689] LustreError: 142272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.107600] LustreError: 142272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4422.076650] LustreError: 142253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640667 with bad export cookie 6881907291472819401 [ 4422.091483] LustreError: 142253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4422.092225] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.798104] Lustre: server umount lustre-MDT0001 complete [ 4431.740240] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 16:37:56 (1788640676) [ 4438.431406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.111686] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4459.296082] LustreError: 146434:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 4485.023271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.027737] Lustre: Skipped 1 previous similar message [ 4491.691527] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4499.799321] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4507.587260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.142427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4513.002420] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4521.851827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.812377] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4534.879835] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4540.890352] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4545.373298] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4545.592951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4549.890813] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4550.182107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4553.185119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.193205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4553.220540] Lustre: Skipped 2 previous similar messages [ 4553.247912] Lustre: Skipped 2 previous similar messages [ 4558.451450] Lustre: server umount lustre-MDT0000 complete [ 4562.431201] LustreError: 146440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640807 with bad export cookie 6881907291472820423 [ 4562.442517] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.728985] Lustre: server umount lustre-MDT0001 complete [ 4576.418621] Lustre: server umount lustre-OST0000 complete [ 4586.022098] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 16:40:30 (1788640830) [ 4593.902913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.631732] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4614.754773] LustreError: 150628:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 4647.775456] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4657.330817] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4664.427650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.972230] LustreError: 150654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.993691] LustreError: 150654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4665.132355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4669.923856] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4678.711070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.648637] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4690.747091] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4696.994872] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4704.023538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4704.300648] Lustre: lustre-OST0001: new disk, initializing [ 4704.304260] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4704.310063] Lustre: 153503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4709.673434] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4710.949870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4710.962917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4711.057185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4719.291726] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4724.679908] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4724.997980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4729.670543] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4729.900422] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4735.461955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4735.476847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.489072] Lustre: Skipped 1 previous similar message [ 4735.495600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4735.500819] Lustre: Skipped 3 previous similar messages [ 4746.207128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4746.339391] Lustre: server umount lustre-OST0000 complete [ 4755.712864] Lustre: server umount lustre-OST0001 complete [ 4761.058902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4774.367269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4774.561101] Lustre: server umount lustre-MDT0000 complete [ 4778.734112] LustreError: 150634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641023 with bad export cookie 6881907291472821375 [ 4778.743718] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.748677] LustreError: 150634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4779.183935] Lustre: server umount lustre-MDT0001 complete [ 4803.849297] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 16:44:08 (1788641048) [ 4805.469494] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4807.208595] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4809.207291] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 16:44:14 (1788641054) [ 4816.347439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.768193] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4816.820541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4817.153620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.161919] Lustre: Skipped 6 previous similar messages [ 4822.038581] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4831.661169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.825169] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4831.870747] Lustre: 157065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4837.064288] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4845.419418] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4851.118212] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4858.259275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4858.464086] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4865.053964] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4867.257044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4872.440575] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4876.170838] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4876.404457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4880.283761] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4880.444691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4882.400695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4882.410162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.443892] Lustre: Skipped 5 previous similar messages [ 4882.456843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4882.468776] Lustre: Skipped 13 previous similar messages [ 4888.395184] Lustre: server umount lustre-OST0000 complete [ 4911.379459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.833922] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.873205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.876864] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4913.911675] Lustre: Skipped 1 previous similar message [ 4919.930213] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4929.839757] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4934.786737] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4934.981763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4939.800712] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4940.099782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4945.624530] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e8d84be1000.ost_server_uuid 50 [ 4947.247786] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e8d84be1000.ost_server_uuid in FULL state after 0 sec [ 4947.438124] LustreError: 158599:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.54@tcp) returned error from blocking AST (req@0000000050ed9f06 x1875525165649280 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000aa068106/0x5f8172383d6885d9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.54@tcp remote: 0x3650aac3e21c8a90 expref: 5 pid: 157073 timeout: 5047 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4947.463202] LustreError: lustre-MDT0000: A client on nid 192.168.203.54@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4947.471943] LustreError: 157061:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.54@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000aa068106/0x5f8172383d6885d9 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.54@tcp remote: 0x3650aac3e21c8a90 expref: 6 pid: 157073 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4953.071709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4958.158224] Lustre: server umount lustre-OST0000 complete [ 4968.830611] Lustre: server umount lustre-MDT0000 complete [ 4970.470973] LustreError: 160212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.507507] LustreError: 160212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4973.237687] LustreError: 157058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641218 with bad export cookie 6881907291472822558 [ 4973.245306] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.253636] LustreError: 157058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4979.602731] Lustre: server umount lustre-MDT0001 complete [ 4989.580955] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4992.446299] Key type lgssc unregistered [ 4992.763903] LNet: 163177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4992.778610] LNetError: 163177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4992.799251] LNet: Removed LNI 192.168.203.154@tcp [ 4993.614787] Key type .llcrypt unregistered [ 4993.616946] Key type ._llcrypt unregistered [ 5018.034681] Key type ._llcrypt registered [ 5018.037185] Key type .llcrypt registered [ 5018.156888] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5019.337352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5019.431414] alg: No test for adler32 (adler32-zlib) [ 5020.558332] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5020.876116] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5022.655337] Key type lgssc registered [ 5024.313450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5032.045658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5032.073947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.500025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5037.738329] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5043.681157] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.699830] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5045.740082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.160756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.654783] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5058.357567] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5064.390316] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.346286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.802305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.211617] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5079.031814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5085.688561] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5091.206185] Lustre: Failing over lustre-MDT0000 [ 5091.480752] Lustre: server umount lustre-MDT0000 complete [ 5092.329114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5092.337264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.359417] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.368817] LustreError: 164590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.377322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.384673] LustreError: 164590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5099.488569] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5099.508261] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5103.796138] LustreError: 164570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.54@tcp arrived at 1788641348 with bad export cookie 3954237123214900216 [ 5104.610661] LustreError: 164584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.626565] LustreError: 164584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5110.752630] Lustre: 163899:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641339/real 1788641339] req@0000000078e548fd x1875526303115136/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788641355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5110.792632] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.243657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.873263] LustreError: 164586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.898507] LustreError: 164586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5121.005724] LustreError: 163897:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a4a0ebd3 x1875526303121152/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5121.387061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.434379] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5126.409671] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5126.646058] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5126.682187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5129.052647] Lustre: lustre-MDT0000: Denying connection for new client 6b596ec1-0a1b-42bf-ba1d-4e898ca6282e (at 192.168.203.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5138.401889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5138.419710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.437528] Lustre: Skipped 1 previous similar message [ 5138.447749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5141.994537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5141.995993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5142.023279] Lustre: Skipped 1 previous similar message [ 5143.705514] Lustre: server umount lustre-OST0000 complete [ 5147.967901] LustreError: 168320:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5147.978295] Lustre: 167668:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5147.985765] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5148.005381] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5148.015630] LustreError: 167668:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000003171054 x1875526303136768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5148.037746] LustreError: 167668:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5148.048103] LustreError: 167668:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5148.057223] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5148.326872] Lustre: server umount lustre-MDT0000 complete [ 5151.467092] LustreError: 164570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641396 with bad export cookie 3954237123214900314 [ 5151.478067] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5151.483729] LustreError: 164570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5151.490329] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.526924] LustreError: 164589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5157.895329] Lustre: server umount lustre-MDT0001 complete [ 5168.873902] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5171.864230] Key type lgssc unregistered [ 5172.184916] LNet: 169071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5172.193726] LNetError: 169071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5172.211878] LNet: Removed LNI 192.168.203.154@tcp [ 5173.137162] Key type .llcrypt unregistered [ 5173.140627] Key type ._llcrypt unregistered [ 5196.205265] Key type ._llcrypt registered [ 5196.207623] Key type .llcrypt registered [ 5196.304563] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5197.127316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5197.411587] alg: No test for adler32 (adler32-zlib) [ 5198.485627] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5198.712549] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5200.383277] Key type lgssc registered [ 5201.362304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5208.360348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5208.386188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.846216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.052459] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5219.310262] LustreError: 170480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.335931] LustreError: 170480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5222.222720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.602978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5226.660850] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5235.905269] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5243.321085] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5250.939619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.368578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.475216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5257.925377] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5266.280024] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.107936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.114921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.129164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.308958] Lustre: server umount lustre-OST0000 complete [ 5279.204345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5279.221037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.230614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.297138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.298161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.091830] Lustre: server umount lustre-MDT0000 complete [ 5288.418424] LustreError: 172971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.452120] LustreError: 172971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5288.605058] LustreError: 173262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641533 with bad export cookie 5706837511868248419 [ 5288.619870] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5298.450922] Lustre: server umount lustre-MDT0001 complete [ 5308.695499] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5311.552217] Key type lgssc unregistered [ 5311.905790] LNet: 174013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5311.927876] LNetError: 174013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5311.950876] LNet: Removed LNI 192.168.203.154@tcp [ 5312.772738] Key type .llcrypt unregistered [ 5312.777257] Key type ._llcrypt unregistered [ 5343.615202] Key type ._llcrypt registered [ 5343.618328] Key type .llcrypt registered [ 5343.753721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5344.620542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5344.732694] alg: No test for adler32 (adler32-zlib) [ 5345.955313] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5346.277215] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5348.007184] Key type lgssc registered [ 5349.146338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5356.852287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5356.885237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.427912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5362.296799] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5368.802111] LustreError: 175710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.822438] LustreError: 175710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5370.410352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.019957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5376.003709] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5384.304874] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5390.163585] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5396.771918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5397.173581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5403.318439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5403.684454] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5410.914476] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5424.096347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.106284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.107811] LustreError: Skipped 1 previous similar message [ 5424.145444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.146830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.152457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.167204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.200904] Lustre: Skipped 1 previous similar message [ 5431.271553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5431.287344] Lustre: Skipped 1 previous similar message [ 5435.872180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5436.046110] Lustre: server umount lustre-OST0000 complete [ 5441.510880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.528597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5441.541425] Lustre: Skipped 1 previous similar message [ 5441.560408] Lustre: Skipped 1 previous similar message [ 5446.391176] Lustre: server umount lustre-MDT0000 complete [ 5446.628552] LustreError: 175711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.640524] LustreError: 175711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5449.522215] LustreError: 175692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641694 with bad export cookie 1276156463998702127 [ 5449.527655] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5449.536211] LustreError: 175692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5449.832608] Lustre: server umount lustre-MDT0001 complete [ 5459.254312] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5461.817621] Key type lgssc unregistered [ 5462.086651] LNet: 179244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5462.103249] LNetError: 179244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5463.143175] LNet: Removed LNI 192.168.203.154@tcp [ 5464.031760] Key type .llcrypt unregistered [ 5464.034114] Key type ._llcrypt unregistered [ 5484.864718] Key type ._llcrypt registered [ 5484.866598] Key type .llcrypt registered [ 5484.966468] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5486.055680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5486.092895] alg: No test for adler32 (adler32-zlib) [ 5487.124851] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5487.358940] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5489.015168] Key type lgssc registered [ 5490.282829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5500.758263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5500.780819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.126793] Lustre: *** cfs_fail_loc=135, val=0*** [ 5502.231641] LustreError: 180743:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5502.241835] LustreError: 180743:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.154@tcp: cfg command failed: rc = -2 [ 5502.253466] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5502.253466] [ 5502.267576] LustreError: MGC192.168.203.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5502.283799] LustreError: 180704:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5502.292513] LustreError: 180704:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5502.302130] LustreError: 180704:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5502.316718] LustreError: 180704:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5502.438496] Lustre: server umount lustre-MDT0000 complete [ 5502.444629] LustreError: 180704:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5509.069701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.506074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5514.144975] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5519.847560] LustreError: 181114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.879354] LustreError: 181114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5524.406443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.941713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5529.296918] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5537.667566] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5542.832067] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5545.443512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5545.453668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.477486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.538185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.538808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.561904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5550.572276] Lustre: Skipped 2 previous similar messages [ 5550.695456] Lustre: server umount lustre-MDT0000 complete [ 5554.037812] LustreError: 181097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641799 with bad export cookie 14374292373279825530 [ 5554.042777] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5554.047076] LustreError: 181097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5554.341047] Lustre: server umount lustre-MDT0001 complete [ 5563.138916] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5565.071940] Key type lgssc unregistered [ 5565.325479] LNet: 183436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5565.338414] LNetError: 183436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5565.352686] LNet: Removed LNI 192.168.203.154@tcp [ 5566.032680] Key type .llcrypt unregistered [ 5566.035917] Key type ._llcrypt unregistered [ 5585.567470] Key type ._llcrypt registered [ 5585.571827] Key type .llcrypt registered [ 5585.675655] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5586.029315] Key type .llcrypt unregistered [ 5586.032589] Key type ._llcrypt unregistered [ 5601.628075] Key type ._llcrypt registered [ 5601.629743] Key type .llcrypt registered [ 5601.685529] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5602.325564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5602.336578] alg: No test for adler32 (adler32-zlib) [ 5603.342389] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5603.531794] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5605.191119] Key type lgssc registered [ 5606.195598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5612.924498] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5612.953520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5628.511933] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5633.631393] LustreError: 185558:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 5660.319284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5666.639147] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5673.901322] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.952619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.473680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.555352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5684.745491] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5685.736675] LustreError: 185584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.752471] LustreError: 185584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5690.850980] LustreError: 185583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.136106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.545424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5698.421754] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5705.694681] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5710.952980] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.622099] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5726.691299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5726.699981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.710547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.716983] Lustre: Skipped 1 previous similar message [ 5731.718982] Lustre: server umount lustre-OST0000 complete [ 5736.929429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.946711] Lustre: Skipped 1 previous similar message [ 5736.951278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5736.957906] Lustre: Skipped 1 previous similar message [ 5741.869821] Lustre: server umount lustre-MDT0000 complete [ 5742.058581] LustreError: 186601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.377498] LustreError: 187338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641990 with bad export cookie 7283904065948882364 [ 5745.390301] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5745.404353] LustreError: 187338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5745.790724] Lustre: server umount lustre-MDT0001 complete [ 5756.090169] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5759.194772] Key type lgssc unregistered [ 5759.535677] LNet: 189572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5759.547648] LNetError: 189572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5759.575468] LNet: Removed LNI 192.168.203.154@tcp [ 5760.452459] Key type .llcrypt unregistered [ 5760.456809] Key type ._llcrypt unregistered [ 5782.915783] Key type ._llcrypt registered [ 5782.918466] Key type .llcrypt registered [ 5783.043176] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5784.037048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5784.117593] alg: No test for adler32 (adler32-zlib) [ 5785.296488] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5785.671046] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5787.449484] Key type lgssc registered [ 5788.757569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5796.794462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5796.807032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.197238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.033336] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5808.624483] LustreError: 190980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.652815] LustreError: 190980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5811.946779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.415500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5816.938689] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5823.953799] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5829.435235] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5836.324555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.686392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.043991] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5843.894088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5849.921945] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5855.604799] Lustre: Failing over lustre-MDT0000 [ 5855.873514] Lustre: server umount lustre-MDT0000 complete [ 5857.928459] LustreError: 190975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.273589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5858.280111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.296781] LustreError: 190980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.314824] LustreError: 190980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5859.323938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.012648] LustreError: 192440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.029326] LustreError: 192440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5868.135965] LustreError: 190977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.163412] LustreError: 190977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5873.859859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.012659] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.201067] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (not set up) [ 5874.287948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.333441] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5878.244417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5878.822706] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5879.801311] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5879.843171] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5879.885724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5882.862367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5894.008268] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5905.891217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.911453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.918829] Lustre: Skipped 1 previous similar message [ 5905.922622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5909.487202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.496494] LustreError: Skipped 1 previous similar message [ 5909.504771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.519058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.917501] Lustre: server umount lustre-OST0000 complete [ 5920.736461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.739221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5920.747682] Lustre: Skipped 1 previous similar message [ 5920.767736] Lustre: Skipped 3 previous similar messages [ 5922.196336] Lustre: server umount lustre-MDT0000 complete [ 5925.768187] LustreError: 190962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642170 with bad export cookie 5629501149404576739 [ 5925.769103] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5925.777595] LustreError: 190962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5926.265455] Lustre: server umount lustre-MDT0001 complete [ 5938.654520] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5941.283419] Key type lgssc unregistered [ 5941.623489] LNet: 195977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5941.647972] LNetError: 195977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5941.667359] LNet: Removed LNI 192.168.203.154@tcp [ 5942.744896] Key type .llcrypt unregistered [ 5942.749517] Key type ._llcrypt unregistered [ 5967.709633] Key type ._llcrypt registered [ 5967.712487] Key type .llcrypt registered [ 5967.834236] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5968.889408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5969.062536] alg: No test for adler32 (adler32-zlib) [ 5970.144540] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5970.363116] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5972.007299] Key type lgssc registered [ 5973.275905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5981.321763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5981.343204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.747443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.836263] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5992.930749] LustreError: 197387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.961705] LustreError: 197387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5995.498682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.909093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6000.538671] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6009.006726] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6014.978873] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6021.999158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.484722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.596561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6028.055392] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6036.091932] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6038.852660] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6044.499726] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6063.261362] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6068.705733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6068.719083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.745146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6071.779487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6071.780630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.804578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.079765] Lustre: server umount lustre-OST0000 complete [ 6077.926069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.948557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.964343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6077.967166] Lustre: Skipped 1 previous similar message [ 6082.017375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.017866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.138890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.153357] Lustre: Skipped 2 previous similar messages [ 6092.255278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6094.485701] Lustre: server umount lustre-MDT0000 complete [ 6096.354980] LustreError: 197387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.376719] LustreError: 197387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6098.044831] LustreError: 197366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642343 with bad export cookie 6587656001516750836 [ 6098.048727] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.060778] LustreError: 197366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6098.310553] Lustre: server umount lustre-MDT0001 complete [ 6109.383278] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6112.079760] Key type lgssc unregistered [ 6112.391948] LNet: 201252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6112.410156] LNetError: 201252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6113.459771] LNet: Removed LNI 192.168.203.154@tcp [ 6114.414891] Key type .llcrypt unregistered [ 6114.417155] Key type ._llcrypt unregistered [ 6134.174368] Key type ._llcrypt registered [ 6134.176940] Key type .llcrypt registered [ 6134.311634] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6135.064643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6135.090382] alg: No test for adler32 (adler32-zlib) [ 6136.373866] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6136.698285] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6138.407786] Key type lgssc registered [ 6139.630799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6147.551506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6147.567334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.082500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.050901] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6159.351579] LustreError: 202513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.375637] LustreError: 202513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6161.985029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.422315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6167.143585] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6174.946093] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6181.242455] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6189.051178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6189.473397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.637064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6195.852921] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6203.713726] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6209.252713] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6218.776212] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6225.137516] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6234.794496] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6243.808613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.815973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.818586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6243.831049] Lustre: Skipped 1 previous similar message [ 6248.937407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6248.941607] Lustre: Skipped 1 previous similar message [ 6249.894377] Lustre: server umount lustre-OST0000 complete [ 6254.049286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.055718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6254.065913] Lustre: Skipped 1 previous similar message [ 6254.074076] Lustre: Skipped 2 previous similar messages [ 6259.170909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6259.188830] Lustre: Skipped 1 previous similar message [ 6259.964740] Lustre: server umount lustre-MDT0000 complete [ 6262.929680] LustreError: 202495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642507 with bad export cookie 15636400407748859640 [ 6262.932232] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6262.934965] LustreError: 202495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6263.210270] Lustre: server umount lustre-MDT0001 complete [ 6275.032041] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6278.698782] Key type lgssc unregistered [ 6279.065111] LNet: 207389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6279.073136] LNetError: 207389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6279.089363] LNet: Removed LNI 192.168.203.154@tcp [ 6279.984523] Key type .llcrypt unregistered [ 6279.986841] Key type ._llcrypt unregistered [ 6305.563799] Key type ._llcrypt registered [ 6305.569438] Key type .llcrypt registered [ 6305.680328] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6306.827771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6306.891820] alg: No test for adler32 (adler32-zlib) [ 6308.001696] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6308.271885] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6309.951179] Key type lgssc registered [ 6311.116931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6319.305916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6319.329534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.818804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.333928] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6330.345374] LustreError: 208798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.363327] LustreError: 208798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6334.913223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.408815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.667751] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6348.594918] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6355.058721] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.444893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.853984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.975591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6369.839285] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6378.725581] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6400.998897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6401.001071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.024444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6401.040653] Lustre: Skipped 1 previous similar message [ 6403.300969] Lustre: server umount lustre-OST0000 complete [ 6411.234318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6411.256837] Lustre: Skipped 1 previous similar message [ 6411.268365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6413.559467] Lustre: server umount lustre-MDT0000 complete [ 6416.353764] LustreError: 208798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.372299] LustreError: 208798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6417.375637] LustreError: 208778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642662 with bad export cookie 13703536125421494467 [ 6417.377367] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6417.385744] LustreError: 208778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6417.935344] Lustre: server umount lustre-MDT0001 complete [ 6429.992699] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6432.809739] Key type lgssc unregistered [ 6433.111231] LNet: 212482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6433.121814] LNetError: 212482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6434.153386] LNet: Removed LNI 192.168.203.154@tcp [ 6435.130112] Key type .llcrypt unregistered [ 6435.132419] Key type ._llcrypt unregistered [ 6455.761599] Key type ._llcrypt registered [ 6455.764874] Key type .llcrypt registered [ 6455.913817] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6457.417358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6457.428102] alg: No test for adler32 (adler32-zlib) [ 6458.622541] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6458.992720] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6460.807266] Key type lgssc registered [ 6462.169348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6470.585962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6470.608191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.108280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.367372] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6482.405412] LustreError: 213745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.442762] LustreError: 213745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6486.565649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.055115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6491.583883] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6498.940067] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6504.360877] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6511.514940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.926436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.038650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6518.672249] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6525.809037] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6541.796207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.797325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.824092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6545.987658] Lustre: server umount lustre-OST0000 complete [ 6552.033848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.050653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.058480] Lustre: Skipped 2 previous similar messages [ 6552.076734] Lustre: Skipped 2 previous similar messages [ 6557.159462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.166491] Lustre: Skipped 1 previous similar message [ 6561.251990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6561.263943] Lustre: Skipped 1 previous similar message [ 6564.319158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6564.491844] Lustre: server umount lustre-MDT0000 complete [ 6566.369848] LustreError: 213745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.388121] LustreError: 213745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6568.079252] LustreError: 213728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642813 with bad export cookie 257324244885565837 [ 6568.085242] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.092564] LustreError: 213728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6568.471720] Lustre: server umount lustre-MDT0001 complete [ 6578.657583] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6581.317750] Key type lgssc unregistered [ 6581.619857] LNet: 217431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6581.626611] LNetError: 217431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6582.703431] LNet: Removed LNI 192.168.203.154@tcp [ 6583.553915] Key type .llcrypt unregistered [ 6583.556650] Key type ._llcrypt unregistered [ 6602.054193] Key type ._llcrypt registered [ 6602.057553] Key type .llcrypt registered [ 6602.160675] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6603.179363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6603.199879] alg: No test for adler32 (adler32-zlib) [ 6604.223121] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6604.437108] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6606.127165] Key type lgssc registered [ 6607.291987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6615.161922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6615.185365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6616.788742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.548196] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6626.280162] LustreError: 218693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.309292] LustreError: 218693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6630.656134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.054141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6635.224124] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6642.507799] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6648.862455] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6655.694654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.914945] LustreError: 220513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6656.124662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.239978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6663.193944] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6671.708882] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6681.242596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6681.299108] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6681.576017] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6687.959886] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6695.776817] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6697.546242] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6709.303681] Lustre: Permanently deactivating lustre-OST0001 [ 6709.321190] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6717.508766] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6727.597239] Lustre: Permanently reactivating lustre-OST0001 [ 6727.617747] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6727.627139] Lustre: Skipped 2 previous similar messages [ 6735.522900] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.544956] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6735.559280] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6735.577914] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6735.625304] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6739.956893] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.963990] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6739.970184] Lustre: Skipped 2 previous similar messages [ 6739.978623] Lustre: Skipped 1 previous similar message [ 6743.727529] Lustre: server umount lustre-OST0001 complete [ 6745.067830] LustreError: 220515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.110465] LustreError: 220515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6748.657421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6748.658348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.676405] LustreError: Skipped 1 previous similar message [ 6748.710102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.182258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.183445] LustreError: 220513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.198610] Lustre: Skipped 2 previous similar messages [ 6750.220515] LustreError: 220513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6754.560190] Lustre: server umount lustre-OST0000 complete [ 6760.426640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.469860] Lustre: Skipped 1 previous similar message [ 6760.483307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.499973] Lustre: Skipped 1 previous similar message [ 6765.574348] LustreError: 218694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.587241] LustreError: 218694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6765.961513] Lustre: server umount lustre-MDT0000 complete [ 6770.377068] LustreError: 218674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643015 with bad export cookie 3143765689081965698 [ 6770.383921] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6770.399675] LustreError: 218674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6770.843413] Lustre: server umount lustre-MDT0001 complete [ 6783.547372] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6786.896951] Key type lgssc unregistered [ 6787.241683] LNet: 223790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6787.251944] LNetError: 223790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6787.280975] LNet: Removed LNI 192.168.203.154@tcp [ 6788.350319] Key type .llcrypt unregistered [ 6788.355079] Key type ._llcrypt unregistered [ 6831.215482] Key type ._llcrypt registered [ 6831.218729] Key type .llcrypt registered [ 6831.325154] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6832.586171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.700753] alg: No test for adler32 (adler32-zlib) [ 6833.846759] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6834.093713] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6835.735149] Key type lgssc registered [ 6836.743986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6845.950817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6845.983584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.367593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6847.407419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6847.530642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.151850] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6863.720797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.965886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6864.026728] Lustre: 225993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6864.269686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6869.427225] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6878.708639] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6885.197819] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6893.020638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6893.278424] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6893.536169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.699483] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6901.938433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6909.265103] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.357954] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6921.314582] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6928.369525] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6938.364517] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6955.121835] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6955.128986] Lustre: Skipped 1 previous similar message [ 6974.794746] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6974.805883] Lustre: Skipped 2 previous similar messages [ 7010.569334] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7010.581575] Lustre: Skipped 3 previous similar messages [ 7077.171904] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7077.176528] Lustre: Skipped 8 previous similar messages [ 7086.055104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7086.069324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.096804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.577392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.577697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7086.609957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.228862] Lustre: server umount lustre-OST0000 complete [ 7094.755052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7094.762064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7094.776226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7096.805217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7096.805794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.624332] Lustre: server umount lustre-MDT0000 complete [ 7104.365391] LustreError: 228836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643349 with bad export cookie 17902886732659196909 [ 7104.368513] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.373482] LustreError: 228836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7104.778185] Lustre: server umount lustre-MDT0001 complete [ 7115.023955] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7117.922806] Key type lgssc unregistered [ 7118.182693] LNet: 230781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7118.191345] LNetError: 230781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7118.209347] LNet: Removed LNI 192.168.203.154@tcp [ 7119.152668] Key type .llcrypt unregistered [ 7119.158082] Key type ._llcrypt unregistered [ 7143.468960] Key type ._llcrypt registered [ 7143.471911] Key type .llcrypt registered [ 7143.597274] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7144.357478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7144.380962] alg: No test for adler32 (adler32-zlib) [ 7145.466255] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 7145.755859] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7147.415156] Key type lgssc registered [ 7148.361946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7155.341464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7155.355499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.699902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.821486] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7166.948770] LustreError: 232193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7166.960490] LustreError: 232193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7169.795365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.178054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7174.317410] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7182.653529] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7188.462825] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7195.371952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7195.727436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.912850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7202.604919] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7210.378747] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7227.875103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7227.889552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.923391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7229.331536] Lustre: server umount lustre-OST0000 complete [ 7236.064667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.078367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.085366] Lustre: Skipped 1 previous similar message [ 7236.098517] Lustre: Skipped 1 previous similar message [ 7239.363104] Lustre: server umount lustre-MDT0000 complete [ 7241.192161] LustreError: 232194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.227920] LustreError: 232194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7243.546180] LustreError: 232174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643488 with bad export cookie 3569150147365273671 [ 7243.559130] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.567157] LustreError: 232174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7254.093471] Lustre: server umount lustre-MDT0001 complete [ 7268.506942] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7271.758058] Key type lgssc unregistered [ 7272.017532] LNet: 235881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7272.026947] LNetError: 235881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7272.045203] LNet: Removed LNI 192.168.203.154@tcp [ 7272.903866] Key type .llcrypt unregistered [ 7272.909747] Key type ._llcrypt unregistered [ 7297.310982] Key type ._llcrypt registered [ 7297.313233] Key type .llcrypt registered [ 7297.429951] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7298.451876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7298.473372] alg: No test for adler32 (adler32-zlib) [ 7299.472716] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 7299.682798] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7301.386868] Key type lgssc registered [ 7302.511973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7571.500394] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7574.252566] Key type lgssc unregistered [ 7574.475966] LNet: 237899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7574.492362] LNetError: 237899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7574.511973] LNet: Removed LNI 192.168.203.154@tcp [ 7575.541404] Key type .llcrypt unregistered [ 7575.544900] Key type ._llcrypt unregistered [ 7601.417258] Key type ._llcrypt registered [ 7601.419712] Key type .llcrypt registered [ 7601.530694] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 7619.389193] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7620.709319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7620.818762] alg: No test for adler32 (adler32-zlib) [ 7621.968668] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 7622.379822] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7624.079181] Key type lgssc registered [ 7625.466334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7661.945552] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7414 sec ========== 17:31:46 (1788643906) [ 7664.008826] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:31:48 (1788643908) === [ 7667.792285] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:31:52 (1788643912) === [ 7710.703061] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7713.565389] Key type lgssc unregistered [ 7713.852669] LNet: 243134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7713.862598] LNetError: 243134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7713.890392] LNet: Removed LNI 192.168.203.154@tcp [ 7714.707686] Key type .llcrypt unregistered [ 7714.713327] Key type ._llcrypt unregistered