[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484198201 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003346] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011065] Yama: becoming mindful. [ 0.012052] SELinux: Initializing. [ 0.013086] *** VALIDATE selinux *** [ 0.021673] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026058] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031066] *** VALIDATE tmpfs *** [ 0.033316] *** VALIDATE proc *** [ 0.035137] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.038084] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041037] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.045008] debug: unmapping init [mem 0xffffffff88259000-0xffffffff88260fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048751] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053021] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056373] rcu: Hierarchical SRCU implementation. [ 0.058559] smp: Bringing up secondary CPUs ... [ 0.059673] x86: Booting SMP configuration: [ 0.060032] .... node #0, CPUs: #1 #2 #3 [ 0.064271] smp: Brought up 1 node, 4 CPUs [ 0.066017] smpboot: Max logical packages: 1 [ 0.067022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.178761] node 0 deferred pages initialised in 109ms [ 0.183050] devtmpfs: initialized [ 0.184314] x86/mm: Memory block size: 128MB [ 0.188052] gcov: version magic: 0x41383552 [ 0.191388] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.198582] pinctrl core: initialized pinctrl subsystem [ 0.200188] [ 0.200866] ************************************************************* [ 0.204018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.206014] ** ** [ 0.208011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.210022] ** ** [ 0.212013] ** This means that this kernel is built to expose internal ** [ 0.214014] ** IOMMU data structures, which may compromise security on ** [ 0.215011] ** your system. ** [ 0.217015] ** ** [ 0.219014] ** If you see this message and you are not debugging the ** [ 0.222012] ** kernel, report this immediately to your vendor! ** [ 0.224014] ** ** [ 0.226016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229011] ************************************************************* [ 0.230715] NET: Registered protocol family 16 [ 0.233487] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.235061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.238068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.240456] cpuidle: using governor menu [ 0.242653] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245340] PCI: Using configuration type 1 for base access [ 0.247119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.255113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.257033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.260179] cryptd: max_cpu_qlen set to 1000 [ 0.265174] ACPI: Added _OSI(Module Device) [ 0.266026] ACPI: Added _OSI(Processor Device) [ 0.268021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.270020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.275590] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.282585] ACPI: Interpreter enabled [ 0.285069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.286015] ACPI: Using IOAPIC for interrupt routing [ 0.287141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.291099] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302340] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.304042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.309144] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.313559] acpiphp: Slot [2] registered [ 0.314236] acpiphp: Slot [5] registered [ 0.316214] acpiphp: Slot [6] registered [ 0.318199] acpiphp: Slot [7] registered [ 0.319188] acpiphp: Slot [8] registered [ 0.321150] acpiphp: Slot [9] registered [ 0.322560] acpiphp: Slot [10] registered [ 0.324134] acpiphp: Slot [3] registered [ 0.326127] acpiphp: Slot [4] registered [ 0.327100] acpiphp: Slot [11] registered [ 0.328098] acpiphp: Slot [12] registered [ 0.330111] acpiphp: Slot [13] registered [ 0.331163] acpiphp: Slot [14] registered [ 0.333111] acpiphp: Slot [15] registered [ 0.335121] acpiphp: Slot [16] registered [ 0.336101] acpiphp: Slot [17] registered [ 0.337121] acpiphp: Slot [18] registered [ 0.339097] acpiphp: Slot [19] registered [ 0.340139] acpiphp: Slot [20] registered [ 0.342115] acpiphp: Slot [21] registered [ 0.343103] acpiphp: Slot [22] registered [ 0.345128] acpiphp: Slot [23] registered [ 0.346121] acpiphp: Slot [24] registered [ 0.347094] acpiphp: Slot [25] registered [ 0.349050] acpiphp: Slot [26] registered [ 0.350172] acpiphp: Slot [27] registered [ 0.352175] acpiphp: Slot [28] registered [ 0.353158] acpiphp: Slot [29] registered [ 0.355189] acpiphp: Slot [30] registered [ 0.357144] acpiphp: Slot [31] registered [ 0.359089] PCI host bridge to bus 0000:00 [ 0.360032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.363040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.365038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.367033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.369032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.372057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.374195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.377043] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.380467] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389734] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.397074] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.401027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.404039] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406769] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.408990] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.412050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.414690] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.419027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.432026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.437032] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.444248] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.489847] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.498028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.543871] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.553028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.562025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.578027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.588477] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.596028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.606025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.623032] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.634662] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.645033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.657021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.678027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.694179] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.701028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.707020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.722028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.735020] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.737270] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.742511] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.745611] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.748322] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.754194] iommu: Default domain type: Passthrough [ 0.756589] SCSI subsystem initialized [ 0.758162] ACPI: bus type USB registered [ 0.760236] usbcore: registered new interface driver usbfs [ 0.762243] usbcore: registered new interface driver hub [ 0.765180] usbcore: registered new device driver usb [ 0.767216] pps_core: LinuxPPS API ver. 1 registered [ 0.770025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.774121] PTP clock support registered [ 0.776177] EDAC MC: Ver: 3.0.0 [ 0.778230] PCI: Using ACPI for IRQ routing [ 0.781293] NetLabel: Initializing [ 0.783014] NetLabel: domain hash size = 128 [ 0.784011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.786095] NetLabel: unlabeled traffic allowed by default [ 0.789062] vgaarb: loaded [ 0.790372] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.792020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799170] clocksource: Switched to clocksource kvm-clock [ 0.923593] VFS: Disk quotas dquot_6.6.0 [ 0.925487] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.928545] *** VALIDATE ramfs *** [ 0.930300] *** VALIDATE hugetlbfs *** [ 0.932219] pnp: PnP ACPI init [ 0.935846] pnp: PnP ACPI: found 6 devices [ 0.972075] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.975919] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.978597] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.981215] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984140] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.987036] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.990652] NET: Registered protocol family 2 [ 0.993540] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.999798] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.004416] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.010153] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.014345] TCP: Hash tables configured (established 65536 bind 65536) [ 1.018246] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.021451] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024743] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027976] NET: Registered protocol family 1 [ 1.031503] RPC: Registered named UNIX socket transport module. [ 1.034078] RPC: Registered udp transport module. [ 1.036051] RPC: Registered tcp transport module. [ 1.037743] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.040247] NET: Registered protocol family 44 [ 1.042023] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.044290] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046532] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049127] PCI: CLS 0 bytes, default 64 [ 1.050772] Unpacking initramfs... [ 2.449099] debug: unmapping init [mem 0xffff96a4bcc54000-0xffff96a4bffbffff] [ 2.454097] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.456434] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.459488] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.990429] Initialise system trusted keyrings [ 4.998028] Key type blacklist registered [ 5.006060] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.045993] zbud: loaded [ 5.059913] *** VALIDATE nfs *** [ 5.061409] *** VALIDATE nfs4 *** [ 5.069298] pstore: using deflate compression [ 5.094275] Platform Keyring initialized [ 5.354743] NET: Registered protocol family 38 [ 5.356415] Key type asymmetric registered [ 5.357800] Asymmetric key parser 'x509' registered [ 5.359677] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.362326] io scheduler mq-deadline registered [ 5.363698] io scheduler kyber registered [ 5.365388] io scheduler bfq registered [ 5.367101] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.370058] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.372023] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.374608] ACPI: Power Button [PWRF] [ 5.381397] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.389112] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.402165] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.410454] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.427658] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.455204] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.491964] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.496941] Non-volatile memory driver v1.3 [ 5.499140] Linux agpgart interface v0.103 [ 5.538184] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.542682] vda: detected capacity change from 0 to 74706944 [ 5.563374] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.566793] vdb: detected capacity change from 0 to 1073741824 [ 5.582316] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.585299] vdc: detected capacity change from 0 to 2621440000 [ 5.605567] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.613534] vdd: detected capacity change from 0 to 2621440000 [ 5.649353] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.655602] vde: detected capacity change from 0 to 4294967296 [ 5.702266] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.707283] vdf: detected capacity change from 0 to 4294967296 [ 5.724947] libphy: Fixed MDIO Bus: probed [ 5.788127] usbcore: registered new interface driver usbserial_generic [ 5.800595] usbserial: USB Serial support registered for generic [ 5.806224] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.814636] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.819815] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.833594] mousedev: PS/2 mouse device common for all mice [ 5.840211] rtc_cmos 00:05: RTC can wake from S4 [ 5.844514] rtc_cmos 00:05: registered as rtc0 [ 5.848780] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.851973] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.858961] intel_pstate: CPU model not supported [ 5.871415] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.879539] hid: raw HID events driver (C) Jiri Kosina [ 5.887276] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.890798] usbcore: registered new interface driver usbhid [ 5.890809] usbhid: USB HID core driver [ 5.890973] drop_monitor: Initializing network drop monitor service [ 5.891188] Initializing XFRM netlink socket [ 5.891558] NET: Registered protocol family 10 [ 5.893351] Segment Routing with IPv6 [ 5.893401] NET: Registered protocol family 17 [ 5.893881] mpls_gso: MPLS GSO support [ 5.960869] RAS: Correctable Errors collector initialized. [ 5.965633] AVX version of gcm_enc/dec engaged. [ 5.968023] AES CTR mode by8 optimization enabled [ 6.154386] sched_clock: Marking stable (6154343607, 0)->(7044026612, -889683005) [ 6.172490] registered taskstats version 1 [ 6.177708] Loading compiled-in X.509 certificates [ 6.182512] zswap: loaded using pool lzo/zbud [ 6.273874] Key type big_key registered [ 6.344295] Key type encrypted registered [ 6.352404] ima: No TPM chip found, activating TPM-bypass! [ 6.360920] ima: Allocated hash algorithm: sha1 [ 6.371031] ima: No architecture policies found [ 6.372181] evm: Initialising EVM extended attributes: [ 6.373358] evm: security.selinux [ 6.374579] evm: security.ima [ 6.376249] evm: security.capability [ 6.380468] evm: HMAC attrs: 0x1 [ 6.384286] rtc_cmos 00:05: setting system clock to 2026-08-14 18:31:43 UTC (1786732303) [ 6.393302] debug: unmapping init [mem 0xffffffff89203000-0xffffffff893fffff] [ 6.397268] debug: unmapping init [mem 0xffffffff87f82000-0xffffffff88258fff] [ 6.409153] Write protecting the kernel read-only data: 28672k [ 6.413418] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 6.416473] debug: unmapping init [mem 0xffffffff86f14000-0xffffffff86ffffff] [ 6.489991] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.503876] systemd[1]: Detected virtualization kvm. [ 6.506592] systemd[1]: Detected architecture x86-64. [ 6.509234] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.562182] systemd[1]: No hostname configured. [ 6.564362] systemd[1]: Set hostname to . [ 6.567084] random: systemd: uninitialized urandom read (16 bytes read) [ 6.570072] systemd[1]: Initializing machine ID from random generator. [ 6.655982] random: ln: uninitialized urandom read (6 bytes read) [ 6.893982] random: systemd: uninitialized urandom read (16 bytes read) [ 6.898510] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.926979] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 6.957579] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.734828] device-mapper: uevent: version 1.0.3 [ 8.761457] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 11.016404] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.141882] virtio_net virtio0 ens2: renamed from eth0 [ 11.315809] scsi host0: ata_piix [ 11.340248] scsi host1: ata_piix [ 11.342582] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.345627] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.895213] random: crng init done [ 15.896579] random: 7 urandom warning(s) missed due to ratelimiting [ 18.451411] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.396503] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.137710] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.409150] SELinux: Disabled at runtime. [ 25.508458] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.528578] systemd[1]: Detected virtualization kvm. [ 25.536325] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.684373] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.694241] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.714851] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.729498] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.748115] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.766985] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.801934] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 28.499211] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 30.080413] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.310262] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.425418] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.109890] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.137205] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.249052] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 37.971688] NFS: Registering the id_resolver key type [ 37.981654] Key type id_resolver registered [ 37.984588] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg221-server login: [ 64.895462] hrtimer: interrupt took 8668019 ns [ 94.839759] libcfs: loading out-of-tree module taints kernel. [ 94.921313] Key type ._llcrypt registered [ 94.925775] Key type .llcrypt registered [ 95.025339] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 112.063361] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 113.636338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.650872] alg: No test for adler32 (adler32-zlib) [ 115.020350] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 116.109862] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 117.992230] Key type lgssc registered [ 119.758272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.369364] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.597152] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 196.549642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.605146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.031104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.113423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.260975] Lustre: lustre-MDT0000: new disk, initializing [ 198.342549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.374294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.242631] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.096218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.248756] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.274744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.293277] Lustre: Skipped 1 previous similar message [ 216.435260] Lustre: lustre-MDT0001: new disk, initializing [ 216.538672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.582273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.612655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.295162] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.789265] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 235.404534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.601746] Lustre: lustre-OST0000: new disk, initializing [ 235.605037] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.608591] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.648836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.701108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.713840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.817408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.206712] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.344556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.444339] Lustre: lustre-OST0001: new disk, initializing [ 255.446861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.451348] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.511436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.142512] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.666459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.676635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.746130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.445469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.356088] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.489842] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing check_logdir /tmp/testlogs/ [ 287.827369] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing yml_node [ 292.438705] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 294.402971] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 297.067875] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 298.508690] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 14:36:35 EDT 2026 [ 313.384259] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 314.709488] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.283053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.285363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.303420] Lustre: Skipped 1 previous similar message [ 333.322336] Lustre: Skipped 3 previous similar messages [ 335.808490] Lustre: server umount lustre-MDT0000 complete [ 338.774500] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786732635 with bad export cookie 5111441738722179816 [ 338.777128] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.795599] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 343.524519] LustreError: 11585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.525077] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.525084] Lustre: Skipped 2 previous similar messages [ 343.525778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.553441] LustreError: 11585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 348.644081] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.644088] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.673450] Lustre: Skipped 2 previous similar messages [ 353.248298] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 353.408421] Lustre: server umount lustre-MDT0001 complete [ 356.889921] Lustre: server umount lustre-OST0000 complete [ 360.883358] Lustre: server umount lustre-OST0001 complete [ 367.650581] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 379.645686] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 413.780299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.960702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.985850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.084187] Lustre: lustre-MDT0000: new disk, initializing [ 414.189837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.203906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.388774] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 430.238143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.425152] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.532610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.536279] Lustre: Skipped 1 previous similar message [ 430.590238] Lustre: lustre-MDT0001: new disk, initializing [ 430.664794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.699025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.705882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.764805] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 445.401312] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 450.981947] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 457.369551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.644447] Lustre: lustre-OST0000: new disk, initializing [ 457.648944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.655404] Lustre: 17051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 457.780387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.207035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 459.217675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.339246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.774421] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 471.373350] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 476.322634] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.632060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.227207] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 482.553387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.838565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.842282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.857142] Lustre: Skipped 1 previous similar message [ 484.861466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.870157] Lustre: Skipped 1 previous similar message [ 489.966798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.034568] Lustre: server umount lustre-OST0000 complete [ 495.077424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.104212] Lustre: Skipped 2 previous similar messages [ 500.195683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.209755] Lustre: Skipped 4 previous similar messages [ 500.846241] Lustre: server umount lustre-MDT0000 complete [ 504.960066] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786732802 with bad export cookie 5111441738722181356 [ 504.969571] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.973275] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 505.412891] Lustre: server umount lustre-MDT0001 complete [ 508.929547] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:40:05 (1786732805) [ 515.534311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.876377] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.895740] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 515.947088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.732496] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 521.194633] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.308372] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.239138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.177987] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 541.810941] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.931435] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 554.996055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 561.606451] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 570.114806] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 582.113665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.116993] LustreError: Skipped 1 previous similar message [ 582.132779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.168933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.154060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.432237] Lustre: server umount lustre-OST0000 complete [ 593.378964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.389574] Lustre: Skipped 2 previous similar messages [ 605.152195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 605.405221] Lustre: server umount lustre-MDT0000 complete [ 608.739711] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.756332] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 609.562957] LustreError: 19298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786732906 with bad export cookie 5111441738722182238 [ 609.583349] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.084591] Lustre: server umount lustre-MDT0001 complete [ 622.022602] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 624.509301] Key type lgssc unregistered [ 624.724846] LNet: 22852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.731728] LNetError: 22852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.739793] LNet: Removed LNI 192.168.202.121@tcp [ 625.498306] Key type .llcrypt unregistered [ 625.500396] Key type ._llcrypt unregistered [ 649.956526] Key type ._llcrypt registered [ 649.958251] Key type .llcrypt registered [ 650.128801] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 651.496582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 651.520018] alg: No test for adler32 (adler32-zlib) [ 652.544600] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 652.792465] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 654.530479] Key type lgssc registered [ 655.621859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 662.696901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.715863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.008239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.193201] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 674.276792] LustreError: 24265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.288851] LustreError: 24265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 676.614683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.888979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 681.269239] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 688.183200] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 694.754378] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.208325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 701.469207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.261695] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 709.622577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 714.586333] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 730.091030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.105459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 730.116209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.120404] Lustre: Skipped 1 previous similar message [ 730.140927] Lustre: Skipped 1 previous similar message [ 730.706266] Lustre: server umount lustre-OST0000 complete [ 735.204469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.212135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.214138] Lustre: Skipped 1 previous similar message [ 735.225870] Lustre: Skipped 1 previous similar message [ 740.261728] Lustre: server umount lustre-MDT0000 complete [ 740.338037] LustreError: 24262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.364597] LustreError: 24262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 744.205851] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786733041 with bad export cookie 1889107574198768126 [ 744.212683] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.217106] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 744.589205] Lustre: server umount lustre-MDT0001 complete [ 757.457484] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 760.341986] Key type lgssc unregistered [ 760.648232] LNet: 28245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.664337] LNetError: 28245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.687641] LNet: Removed LNI 192.168.202.121@tcp [ 761.609992] Key type .llcrypt unregistered [ 761.614707] Key type ._llcrypt unregistered [ 787.209271] Key type ._llcrypt registered [ 787.213383] Key type .llcrypt registered [ 787.311516] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 788.768612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.891067] alg: No test for adler32 (adler32-zlib) [ 789.900569] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 790.104976] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 791.784247] Key type lgssc registered [ 792.884689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 799.281827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 799.294580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.635215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.581684] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 809.954611] LustreError: 29661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 809.971035] LustreError: 29661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 813.414589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.849228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 819.735429] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 829.810214] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 835.266325] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 850.903206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 851.142268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.088733] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 864.650204] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 870.609704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 876.002187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.011207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 876.016969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 876.019148] Lustre: Skipped 1 previous similar message [ 878.116201] Lustre: server umount lustre-OST0000 complete [ 884.196427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.207387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.218578] Lustre: Skipped 2 previous similar messages [ 887.963915] Lustre: server umount lustre-MDT0000 complete [ 889.313605] LustreError: 29655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.337763] LustreError: 29655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 892.673647] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786733189 with bad export cookie 10897510928426602556 [ 892.697772] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.205730] Lustre: server umount lustre-MDT0001 complete [ 906.017781] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 909.105548] Key type lgssc unregistered [ 909.500581] LNet: 33540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.513445] LNetError: 33540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.529397] LNet: Removed LNI 192.168.202.121@tcp [ 910.362257] Key type .llcrypt unregistered [ 910.365219] Key type ._llcrypt unregistered [ 935.524180] Key type ._llcrypt registered [ 935.534952] Key type .llcrypt registered [ 935.645269] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 936.726807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.770630] alg: No test for adler32 (adler32-zlib) [ 937.913240] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 938.191409] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 939.840239] Key type lgssc registered [ 941.201915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.335907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.349842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.726952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.115412] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 958.946670] LustreError: 34954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.957110] LustreError: 34954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 961.931105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.300699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.193622] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 975.266591] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 981.471965] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 989.638488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.271319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.376505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 997.365640] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1006.506206] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1017.832646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.837433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.840068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.854126] Lustre: Skipped 1 previous similar message [ 1022.184981] Lustre: server umount lustre-OST0000 complete [ 1028.065932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.086117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.093558] Lustre: Skipped 2 previous similar messages [ 1033.187338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.191123] Lustre: Skipped 1 previous similar message [ 1038.306902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.323624] Lustre: Skipped 1 previous similar message [ 1040.352454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.557476] Lustre: server umount lustre-MDT0000 complete [ 1043.429986] LustreError: 35959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.451900] LustreError: 35959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1044.697454] LustreError: 34937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786733341 with bad export cookie 11548352169215695217 [ 1044.705947] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.727197] LustreError: 34937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1044.976297] Lustre: server umount lustre-MDT0001 complete [ 1053.951721] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1056.295282] Key type lgssc unregistered [ 1056.580298] LNet: 38487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.586868] LNetError: 38487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.601758] LNet: Removed LNI 192.168.202.121@tcp [ 1057.467576] Key type .llcrypt unregistered [ 1057.472788] Key type ._llcrypt unregistered [ 1082.290400] Key type ._llcrypt registered [ 1082.293603] Key type .llcrypt registered [ 1082.398645] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1083.567332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.617156] alg: No test for adler32 (adler32-zlib) [ 1084.728934] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 1084.899876] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1086.553417] Key type lgssc registered [ 1087.615681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.403745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1094.417143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.773010] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.444820] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1105.902038] LustreError: 39902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.927058] LustreError: 39902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1107.759524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.028082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1111.600484] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1119.121749] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1125.059532] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1132.034760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.364828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.536654] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1141.646624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1146.163212] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1153.935443] Lustre: lustre-OST0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 1154.028724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.037360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.092727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.102258] Lustre: Skipped 1 previous similar message [ 1157.102516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.109187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.033557] Lustre: lustre-OST0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 1159.693024] Lustre: server umount lustre-OST0000 complete [ 1182.437758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.440527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1182.452943] Lustre: Skipped 1 previous similar message [ 1182.468156] Lustre: Skipped 1 previous similar message [ 1186.860953] Lustre: server umount lustre-MDT0000 complete [ 1187.810294] LustreError: 40968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.840707] LustreError: 40968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1191.796287] LustreError: 39883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786733488 with bad export cookie 1103716763058647246 [ 1191.819191] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.240222] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1206.620125] Lustre: server umount lustre-MDT0001 complete [ 1218.701446] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1221.736261] Key type lgssc unregistered [ 1222.103238] LNet: 43611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.115830] LNetError: 43611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.130169] LNet: Removed LNI 192.168.202.121@tcp [ 1223.327582] Key type .llcrypt unregistered [ 1223.329576] Key type ._llcrypt unregistered [ 1252.044591] Key type ._llcrypt registered [ 1252.048308] Key type .llcrypt registered [ 1252.161835] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1253.729763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1253.771064] alg: No test for adler32 (adler32-zlib) [ 1254.931324] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 1255.273743] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1257.072201] Key type lgssc registered [ 1258.355808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.818924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.862770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.616691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.771570] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1277.922226] LustreError: 45031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.944168] LustreError: 45031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1281.788615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.047762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.613700] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1294.033224] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1299.457715] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1306.182492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1306.542149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.097592] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1313.784428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1321.787471] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1328.099930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1328.109925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.131573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.123031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.124492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.153268] Lustre: Skipped 1 previous similar message [ 1329.163526] Lustre: Skipped 3 previous similar messages [ 1329.722761] Lustre: server umount lustre-MDT0000 complete [ 1333.619195] LustreError: 46871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.860439] LustreError: 45010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786733631 with bad export cookie 9620463163773146140 [ 1334.870461] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.874595] LustreError: 45010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1338.737257] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.759348] LustreError: 45025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1338.771849] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 1339.362711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.466781] Lustre: server umount lustre-MDT0001 complete [ 1363.297862] Lustre: server umount lustre-OST0000 complete [ 1383.166422] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1387.755960] Key type lgssc unregistered [ 1388.253769] LNet: 48889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.266995] LNetError: 48889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.280336] LNet: Removed LNI 192.168.202.121@tcp [ 1389.157147] Key type .llcrypt unregistered [ 1389.159075] Key type ._llcrypt unregistered [ 1418.399439] Key type ._llcrypt registered [ 1418.404429] Key type .llcrypt registered [ 1418.533926] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1419.694375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1419.747681] alg: No test for adler32 (adler32-zlib) [ 1420.954917] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 1421.171034] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1422.928801] Key type lgssc registered [ 1424.211898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.164263] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.189359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1447.712543] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1452.832951] LustreError: 50278:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 1479.522123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.614941] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1494.277173] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1658.685593] Lustre: server umount lustre-OST0000 complete [ 1675.733963] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1679.018279] Key type lgssc unregistered [ 1679.334230] LNet: 51953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.342225] LNetError: 51953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.360076] LNet: Removed LNI 192.168.202.121@tcp [ 1681.254873] Key type .llcrypt unregistered [ 1681.258099] Key type ._llcrypt unregistered [ 1707.157927] Key type ._llcrypt registered [ 1707.161732] Key type .llcrypt registered [ 1707.258599] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1708.733958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.821454] alg: No test for adler32 (adler32-zlib) [ 1709.980708] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 1710.255140] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1711.920521] Key type lgssc registered [ 1712.981741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.743239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.758375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.208472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.491547] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1731.589320] LustreError: 53367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.613447] LustreError: 53367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1733.540961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.293902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.050597] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1751.157686] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.606311] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1767.045034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1767.398560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.977976] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1781.006973] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1784.804652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1784.812152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.827631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.614475] Lustre: server umount lustre-OST0000 complete [ 1794.978575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.990560] Lustre: Skipped 1 previous similar message [ 1794.990678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.006493] Lustre: Skipped 1 previous similar message [ 1800.164383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.291535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.302873] Lustre: Skipped 2 previous similar messages [ 1808.352115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1808.660458] Lustre: server umount lustre-MDT0000 complete [ 1810.401658] LustreError: 53363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.415739] LustreError: 53363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1812.198669] LustreError: 53347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786734109 with bad export cookie 13746731921778747497 [ 1812.203756] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1812.207234] LustreError: 53347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1812.571475] Lustre: server umount lustre-MDT0001 complete [ 1823.525577] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1825.554567] Key type lgssc unregistered [ 1825.786735] LNet: 56913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.793686] LNetError: 56913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.810389] LNet: Removed LNI 192.168.202.121@tcp [ 1826.472820] Key type .llcrypt unregistered [ 1826.476621] Key type ._llcrypt unregistered [ 1851.176549] Key type ._llcrypt registered [ 1851.178736] Key type .llcrypt registered [ 1851.255501] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1852.503469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.628225] alg: No test for adler32 (adler32-zlib) [ 1853.655741] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 1853.889485] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1855.592927] Key type lgssc registered [ 1856.766372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1864.278935] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1864.318355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.905937] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1885.024349] LustreError: 58301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 1911.713283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.133749] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1926.869871] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1933.834348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.177582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.256702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1938.656202] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1939.427382] LustreError: 58326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.441646] LustreError: 58326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1944.551246] LustreError: 58327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.614269] LustreError: 58326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.644894] LustreError: 58326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1947.586897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.008735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.864330] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1962.300604] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1967.868331] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1970.146287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1970.154446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.175295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.734630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.743355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.749950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.786563] Lustre: Skipped 1 previous similar message [ 1976.035395] Lustre: server umount lustre-OST0000 complete [ 1999.332141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1999.336463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.345086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.441403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.446912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.469313] Lustre: Skipped 1 previous similar message [ 2008.551803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.573062] Lustre: Skipped 1 previous similar message [ 2013.152142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2013.615548] Lustre: server umount lustre-MDT0000 complete [ 2013.670413] LustreError: 61030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.686247] LustreError: 61030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2017.450595] LustreError: 58308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786734314 with bad export cookie 14743540788243858400 [ 2017.454893] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.464308] LustreError: 58308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2023.815319] Lustre: server umount lustre-MDT0001 complete [ 2032.785697] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2035.004300] Key type lgssc unregistered [ 2035.327797] LNet: 62047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.334455] LNetError: 62047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.360799] LNet: Removed LNI 192.168.202.121@tcp [ 2036.036586] Key type .llcrypt unregistered [ 2036.039137] Key type ._llcrypt unregistered [ 2056.498827] Key type ._llcrypt registered [ 2056.502174] Key type .llcrypt registered [ 2056.607554] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2057.464730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.608629] alg: No test for adler32 (adler32-zlib) [ 2058.753664] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2058.944895] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2060.608484] Key type lgssc registered [ 2061.949667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.245737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.284451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.790989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.252641] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2080.239201] LustreError: 63439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.255575] LustreError: 63439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2084.937616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2085.229823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.895841] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2100.264183] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.781223] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.825714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2117.299552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.995938] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2131.092143] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2140.860992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2151.404108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.420632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2151.426541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2151.427202] Lustre: Skipped 1 previous similar message [ 2151.446170] Lustre: Skipped 1 previous similar message [ 2155.420835] Lustre: server umount lustre-OST0000 complete [ 2161.633843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.640117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.650531] Lustre: Skipped 1 previous similar message [ 2165.765145] Lustre: server umount lustre-MDT0000 complete [ 2166.753407] LustreError: 63439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.764524] LustreError: 63439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2169.050334] LustreError: 63420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786734466 with bad export cookie 5681897061927331434 [ 2169.066126] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.413435] Lustre: server umount lustre-MDT0001 complete [ 2179.378873] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2181.461623] Key type lgssc unregistered [ 2181.673155] LNet: 66997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.676183] LNetError: 66997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.688747] LNet: Removed LNI 192.168.202.121@tcp [ 2182.353435] Key type .llcrypt unregistered [ 2182.355557] Key type ._llcrypt unregistered [ 2205.445160] Key type ._llcrypt registered [ 2205.446619] Key type .llcrypt registered [ 2205.547638] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2206.468426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2206.614313] alg: No test for adler32 (adler32-zlib) [ 2207.780547] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2208.116850] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2209.840236] Key type lgssc registered [ 2211.047323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2220.580140] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 15:08:37 (1786734517) [ 2227.329181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2227.343147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.658793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.591584] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2237.936592] LustreError: 68700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.951571] LustreError: 68700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2241.976887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.293866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.422769] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2253.543537] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2259.352445] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2264.841984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.184733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.694196] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2273.393701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2281.269465] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2287.811719] Lustre: Failing over lustre-MDT0000 [ 2288.609351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2288.621231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.640035] LustreError: 69428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.135900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.152674] Lustre: Skipped 1 previous similar message [ 2290.163241] Lustre: server umount lustre-MDT0000 complete [ 2294.134960] LustreError: 68697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.170206] LustreError: 68697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2299.251683] LustreError: 68697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.272450] LustreError: 68697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2300.108060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.296596] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.481990] Lustre: *** cfs_fail_loc=135, val=0*** [ 2300.588654] LustreError: 71787:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2300.598560] LustreError: 71787:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.121@tcp: cfg command failed: rc = -2 [ 2300.608945] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2300.608945] [ 2300.623282] LustreError: MGC192.168.202.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2300.646198] LustreError: 71775:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2300.652524] LustreError: 71775:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2300.667665] LustreError: 71775:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2300.674964] LustreError: 71775:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2300.708146] Lustre: server umount lustre-MDT0000 complete [ 2300.711273] LustreError: 71775:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2304.366187] LustreError: 68695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.386740] LustreError: 68695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2308.241431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.384943] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.599768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.678561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2309.489126] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2313.726764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2313.803863] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2313.879328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2313.989052] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2323.942964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.945115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.958705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.981613] Lustre: Skipped 1 previous similar message [ 2327.813433] Lustre: server umount lustre-OST0000 complete [ 2334.178974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2334.181697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.183896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.183904] Lustre: Skipped 1 previous similar message [ 2334.214541] Lustre: Skipped 1 previous similar message [ 2337.503061] Lustre: server umount lustre-MDT0000 complete [ 2339.298170] LustreError: 69428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.321444] LustreError: 69428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2340.611929] LustreError: 68680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786734637 with bad export cookie 17564230072152205707 [ 2340.614539] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2340.619033] LustreError: 68680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2340.839754] Lustre: server umount lustre-MDT0001 complete [ 2353.155243] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2356.187549] Key type lgssc unregistered [ 2356.466696] LNet: 73566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.472425] LNetError: 73566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.484916] LNet: Removed LNI 192.168.202.121@tcp [ 2357.299703] Key type .llcrypt unregistered [ 2357.303541] Key type ._llcrypt unregistered [ 2383.193088] Key type ._llcrypt registered [ 2383.195327] Key type .llcrypt registered [ 2383.294267] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2384.282069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2384.330526] alg: No test for adler32 (adler32-zlib) [ 2385.367098] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2385.560201] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2387.208141] Key type lgssc registered [ 2388.157049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2396.439528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2396.465172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.929187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.052101] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2407.405476] LustreError: 74992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.423960] LustreError: 74992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2412.090802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.408471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2418.216401] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2427.318564] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2432.847186] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2440.640844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2441.032868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.134475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2448.033652] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2457.616799] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2465.170955] Lustre: Failing over lustre-MDT0000 [ 2465.374050] Lustre: server umount lustre-MDT0000 complete [ 2465.693994] LustreError: 74988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.819763] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.851484] Lustre: Skipped 1 previous similar message [ 2467.853570] LustreError: 74993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.898701] LustreError: 74993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2470.774200] LustreError: 74987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.896051] LustreError: 74989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.925339] LustreError: 74989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2477.834681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.956650] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2478.051304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2478.064173] Lustre: Skipped 2 previous similar messages [ 2478.120708] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2478.245644] LustreError: 78083:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2478.253123] LustreError: 78083:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.121@tcp: cfg command failed: rc = -9 [ 2478.265925] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2478.265925] [ 2478.278446] LustreError: MGC192.168.202.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2478.292610] LustreError: 78071:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2478.304873] LustreError: 78071:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2478.329167] LustreError: 78071:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2478.340862] LustreError: 78071:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2478.398080] Lustre: server umount lustre-MDT0000 complete [ 2478.401086] LustreError: 78071:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2486.139467] LustreError: 74987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.161512] LustreError: 74987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2486.612761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.898132] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.912256] LustreError: 74984:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2487.405536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.528117] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2491.247682] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2492.245698] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2492.399363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2492.432494] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2492.494515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2502.628404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.638326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2502.655201] Lustre: Skipped 2 previous similar messages [ 2506.141903] Lustre: server umount lustre-OST0000 complete [ 2512.866331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2512.875078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.882617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2512.882630] Lustre: Skipped 1 previous similar message [ 2512.932026] Lustre: Skipped 1 previous similar message [ 2517.543534] Lustre: server umount lustre-MDT0000 complete [ 2517.988549] LustreError: 76121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.026632] LustreError: 76121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2521.982334] LustreError: 74974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786734819 with bad export cookie 5809792835373321260 [ 2521.991899] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.262908] Lustre: server umount lustre-MDT0001 complete [ 2534.902168] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2537.228359] Key type lgssc unregistered [ 2537.507342] LNet: 79865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.520881] LNetError: 79865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.555373] LNet: Removed LNI 192.168.202.121@tcp [ 2538.308846] Key type .llcrypt unregistered [ 2538.311474] Key type ._llcrypt unregistered [ 2553.524277] Key type ._llcrypt registered [ 2553.530753] Key type .llcrypt registered [ 2553.651961] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2553.922800] Key type .llcrypt unregistered [ 2553.925312] Key type ._llcrypt unregistered [ 2580.764320] Key type ._llcrypt registered [ 2580.772063] Key type .llcrypt registered [ 2581.017503] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2582.987699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.009596] alg: No test for adler32 (adler32-zlib) [ 2584.276662] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2584.539477] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2586.288290] Key type lgssc registered [ 2587.530730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2597.273689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2597.293587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.875254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.433425] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2608.102160] LustreError: 81769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.118750] LustreError: 81769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2612.908366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.213053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2617.232913] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2625.789346] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2632.259930] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2639.033647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.381726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.513352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2645.581924] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2654.214579] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2669.026296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.026537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.033021] LustreError: Skipped 1 previous similar message [ 2669.034894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.072231] Lustre: Skipped 1 previous similar message [ 2674.148528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.155737] Lustre: Skipped 1 previous similar message [ 2674.934510] Lustre: server umount lustre-OST0000 complete [ 2679.266261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.285947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.294603] Lustre: Skipped 2 previous similar messages [ 2684.386473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.391717] Lustre: Skipped 1 previous similar message [ 2684.936707] Lustre: server umount lustre-MDT0000 complete [ 2688.613821] LustreError: 81749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786734985 with bad export cookie 10365771058646852263 [ 2688.621960] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.624450] LustreError: 81749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2689.161286] Lustre: server umount lustre-MDT0001 complete [ 2699.888353] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2702.276650] Key type lgssc unregistered [ 2702.555490] LNet: 85324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.563725] LNetError: 85324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.576790] LNet: Removed LNI 192.168.202.121@tcp [ 2703.617603] Key type .llcrypt unregistered [ 2703.619099] Key type ._llcrypt unregistered [ 2726.275155] Key type ._llcrypt registered [ 2726.276817] Key type .llcrypt registered [ 2726.371374] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2727.650219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2727.720379] alg: No test for adler32 (adler32-zlib) [ 2728.874450] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2729.191268] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2730.928240] Key type lgssc registered [ 2732.161691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.823416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2739.848163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.336238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.599447] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2751.468080] LustreError: 86735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.491835] LustreError: 86735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2754.907767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.293649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2759.127917] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2766.841241] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2772.908656] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2779.369407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.664929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.831733] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2786.886877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2792.809877] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2800.796440] Lustre: server umount lustre-OST0000 complete [ 2806.753940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2806.772241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.805186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.396395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.417111] Lustre: Skipped 3 previous similar messages [ 2815.566425] Lustre: server umount lustre-MDT0000 complete [ 2817.509033] Lustre: 86049:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786735098/real 1786735098] req@0000000078769eda x1873527466648960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786735114 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2817.521453] LustreError: 86730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.544528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.544543] Lustre: Skipped 1 previous similar message [ 2817.610978] LustreError: 86730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2818.528546] Lustre: 86050:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786735099/real 1786735099] req@000000002a99e659 x1873527466649984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786735115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2820.134459] LustreError: 86715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786735117 with bad export cookie 15526468203532002648 [ 2820.150797] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2820.156557] LustreError: 86715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2820.496750] Lustre: server umount lustre-MDT0001 complete [ 2830.925922] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2833.542256] Key type lgssc unregistered [ 2833.831659] LNet: 90268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.838280] LNetError: 90268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.851782] LNet: Removed LNI 192.168.202.121@tcp [ 2834.704322] Key type .llcrypt unregistered [ 2834.706481] Key type ._llcrypt unregistered [ 2857.718473] Key type ._llcrypt registered [ 2857.736386] Key type .llcrypt registered [ 2857.963873] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2859.653784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2859.723893] alg: No test for adler32 (adler32-zlib) [ 2861.609442] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2862.037156] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2863.793418] Key type lgssc registered [ 2866.157116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2875.831825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2875.876288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.169347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.856575] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2888.681374] LustreError: 91683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.717927] LustreError: 91683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2893.798930] LustreError: 91682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.258725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.821091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2900.683753] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2908.350976] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2914.375483] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2921.036206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.583911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.663665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2928.128731] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2937.688164] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2954.725402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.726791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.735707] LustreError: Skipped 1 previous similar message [ 2954.766341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.748469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.755817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.874418] LustreError: 93500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.885072] LustreError: 93500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2960.955700] Lustre: server umount lustre-OST0000 complete [ 2965.986395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.988118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.020719] Lustre: Skipped 1 previous similar message [ 2966.031198] Lustre: Skipped 1 previous similar message [ 2970.560406] Lustre: server umount lustre-MDT0000 complete [ 2971.110188] LustreError: 91682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.154970] LustreError: 91682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2975.258297] LustreError: 91663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786735272 with bad export cookie 14437062820053495892 [ 2975.262066] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.277986] LustreError: 91663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2975.775606] Lustre: server umount lustre-MDT0001 complete [ 2989.061350] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2993.040842] Key type lgssc unregistered [ 2993.362864] LNet: 95237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2993.369438] LNetError: 95237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2993.382814] LNet: Removed LNI 192.168.202.121@tcp [ 2994.310708] Key type .llcrypt unregistered [ 2994.313912] Key type ._llcrypt unregistered [ 3017.676815] Key type ._llcrypt registered [ 3017.684022] Key type .llcrypt registered [ 3017.802836] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3018.765679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.798278] alg: No test for adler32 (adler32-zlib) [ 3020.188804] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 3020.502529] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3022.217943] Key type lgssc registered [ 3023.412977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3032.133307] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3032.150386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.776617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3052.896514] LustreError: 96626:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 3079.648250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.351679] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3094.425305] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3100.827158] Lustre: server umount lustre-OST0000 complete [ 3113.401450] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:23:29 (1786735409) [ 3120.168276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.567900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.098440] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3125.733823] LustreError: 98267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.760183] LustreError: 98267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3130.852278] LustreError: 98266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.283247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.788405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3138.481067] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3146.351139] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3152.685734] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3158.934525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.368851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.236695] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3165.573480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3172.247652] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3186.146326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.152428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.171603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.232323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.247520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.267504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.869716] Lustre: server umount lustre-OST0000 complete [ 3199.457694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.462605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.465378] Lustre: Skipped 1 previous similar message [ 3199.472423] Lustre: Skipped 2 previous similar messages [ 3202.617615] Lustre: server umount lustre-MDT0000 complete [ 3204.581647] LustreError: 98267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.622404] LustreError: 98267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3207.044424] LustreError: 98248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786735504 with bad export cookie 13845639133349008485 [ 3207.050471] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3207.062554] LustreError: 98248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3207.692045] Lustre: server umount lustre-MDT0001 complete [ 3217.899626] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3220.271226] Key type lgssc unregistered [ 3220.551568] LNet: 101959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3220.557622] LNetError: 101959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3220.598531] LNet: Removed LNI 192.168.202.121@tcp [ 3221.669453] Key type .llcrypt unregistered [ 3221.676069] Key type ._llcrypt unregistered [ 3263.112571] Key type ._llcrypt registered [ 3263.117020] Key type .llcrypt registered [ 3263.208572] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3264.663297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3264.698329] alg: No test for adler32 (adler32-zlib) [ 3265.734711] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 3265.894922] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3267.528193] Key type lgssc registered [ 3268.869318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3275.964517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3276.030033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.664108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.556718] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3287.012248] LustreError: 104146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.035623] LustreError: 104146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3292.147491] LustreError: 104147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.408104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.983502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3299.385971] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3308.449605] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3313.475380] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3321.619451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3322.124180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.127328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3332.692517] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3342.502550] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3354.084581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3354.092654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.097813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.099213] LustreError: Skipped 1 previous similar message [ 3354.135454] Lustre: Skipped 1 previous similar message [ 3358.305220] Lustre: server umount lustre-OST0000 complete [ 3364.323286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.357052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3364.370784] Lustre: Skipped 2 previous similar messages [ 3368.053497] Lustre: server umount lustre-MDT0000 complete [ 3369.450891] LustreError: 104146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.247314] LustreError: 104881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786735669 with bad export cookie 18180760746720280633 [ 3372.257417] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.271697] LustreError: 104881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.657873] Lustre: server umount lustre-MDT0001 complete [ 3383.655325] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3385.645829] Key type lgssc unregistered [ 3385.906408] LNet: 107682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3385.913831] LNetError: 107682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3385.950890] LNet: Removed LNI 192.168.202.121@tcp [ 3386.747438] Key type .llcrypt unregistered [ 3386.749971] Key type ._llcrypt unregistered [ 3405.722329] Key type ._llcrypt registered [ 3405.723906] Key type .llcrypt registered [ 3405.889600] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3407.587938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3407.721097] alg: No test for adler32 (adler32-zlib) [ 3408.747994] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 3409.077232] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3410.737063] Key type lgssc registered [ 3412.028675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3420.496246] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3420.513712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3436.064522] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3441.184462] LustreError: 108926:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 3467.872293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.815784] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3484.286153] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3492.461796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.628621] LustreError: MGC192.168.202.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3492.636114] LustreError: 109951:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3492.640635] LustreError: 109951:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3492.647672] LustreError: 109951:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3492.708896] Lustre: server umount lustre-MDT0000 complete [ 3492.711670] LustreError: 109951:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3512.292111] Lustre: 108263:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786735793/real 1786735793] req@00000000b9066eeb x1873528179658112/t0(0) o400->MGC192.168.202.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786735809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3512.354880] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3512.364407] LustreError: 108938:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3532.939952] Lustre: server umount lustre-OST0000 complete [ 3542.033635] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3551.698089] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3585.229401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.478838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3585.506496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3585.603303] Lustre: lustre-MDT0000: new disk, initializing [ 3585.716956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.745288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3590.134481] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3600.071086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.183715] Lustre: 112745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3600.228285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3600.230640] Lustre: Skipped 1 previous similar message [ 3600.278762] Lustre: lustre-MDT0001: new disk, initializing [ 3600.360595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3600.393505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3600.411445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3604.562328] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3612.114287] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.272564] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3622.439550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.652318] Lustre: lustre-OST0000: new disk, initializing [ 3622.655837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3622.661825] Lustre: 114714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3622.765625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.571764] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3630.693753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3630.707765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3630.865065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3636.150735] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3641.038710] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3641.295286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3645.810800] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3646.045254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3651.554799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3651.556586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.579635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.589539] Lustre: Skipped 1 previous similar message [ 3654.718956] Lustre: server umount lustre-OST0000 complete [ 3661.797637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.810080] Lustre: Skipped 1 previous similar message [ 3661.812924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3661.816693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.821894] LustreError: Skipped 1 previous similar message [ 3661.833563] Lustre: Skipped 1 previous similar message [ 3665.053832] Lustre: server umount lustre-MDT0000 complete [ 3669.517823] LustreError: 112738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786735966 with bad export cookie 14552523074313561356 [ 3669.524064] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3669.539151] LustreError: 112738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.203764] Lustre: server umount lustre-MDT0001 complete [ 3680.730618] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:32:56 (1786735976) [ 3684.187576] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3702.388215] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3717.455328] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3757.982884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.233982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3758.271779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3758.370203] Lustre: lustre-MDT0000: new disk, initializing [ 3758.475072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.493245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3762.594564] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3772.786362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.877574] Lustre: 119632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3772.935702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3772.944888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3772.951496] Lustre: Skipped 1 previous similar message [ 3773.030858] Lustre: lustre-MDT0001: new disk, initializing [ 3773.143903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3773.194272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3773.207965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3776.959831] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3786.711621] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.566740] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3799.336528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.563326] Lustre: lustre-OST0000: new disk, initializing [ 3799.567324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3799.572517] Lustre: 121600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3799.655523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.506582] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3808.335600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3808.355967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3808.417500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3815.461349] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3820.189523] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3820.422634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3824.843807] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3825.207900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3829.217301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.220732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.224090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.235530] Lustre: Skipped 2 previous similar messages [ 3833.464571] Lustre: server umount lustre-OST0000 complete [ 3839.459172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.467255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.470030] Lustre: Skipped 1 previous similar message [ 3839.484522] Lustre: Skipped 2 previous similar messages [ 3849.699382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3849.707049] Lustre: Skipped 3 previous similar messages [ 3851.233268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3851.443498] Lustre: server umount lustre-MDT0000 complete [ 3854.571407] LustreError: 119622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736151 with bad export cookie 14552523074313562238 [ 3854.584965] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.815372] Lustre: server umount lustre-MDT0001 complete [ 3862.813218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.400843] LustreError: 123816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.420810] LustreError: 123816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3863.461408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.619341] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3868.646264] LustreError: 123817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.770380] LustreError: 123816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.021652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.171596] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3885.694703] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3891.457986] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3897.953622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3905.718158] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3916.454261] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3924.666855] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3931.106965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3931.114590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.142620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.682027] Lustre: server umount lustre-OST0000 complete [ 3941.857526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.868813] LustreError: Skipped 1 previous similar message [ 3941.875567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.893513] Lustre: Skipped 1 previous similar message [ 3947.632642] Lustre: server umount lustre-MDT0000 complete [ 3950.055019] LustreError: 124544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.114968] LustreError: 124544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3953.164908] LustreError: 123796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736250 with bad export cookie 14552523074313563113 [ 3953.171593] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.179549] LustreError: 123796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3953.895305] Lustre: server umount lustre-MDT0001 complete [ 3970.123476] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3975.231681] Key type lgssc unregistered [ 3975.774709] LNet: 127448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3975.790282] LNetError: 127448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3975.818744] LNet: Removed LNI 192.168.202.121@tcp [ 3977.286287] Key type .llcrypt unregistered [ 3977.288635] Key type ._llcrypt unregistered [ 3997.459858] Key type ._llcrypt registered [ 3997.461722] Key type .llcrypt registered [ 3997.508927] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4013.485419] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4014.344684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4014.461678] alg: No test for adler32 (adler32-zlib) [ 4015.488476] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 4015.729829] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4017.416177] Key type lgssc registered [ 4019.026228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4052.931656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4052.957864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.146282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4054.165931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4054.233699] Lustre: lustre-MDT0000: new disk, initializing [ 4054.332993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.357136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4058.404732] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4069.900837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.013661] Lustre: 130600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4070.074307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4070.079080] Lustre: Skipped 1 previous similar message [ 4070.181837] Lustre: lustre-MDT0001: new disk, initializing [ 4070.308583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4070.389166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4070.397161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4076.301912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4086.795437] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.829465] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4098.444729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4098.741796] Lustre: lustre-OST0000: new disk, initializing [ 4098.745222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4098.753296] Lustre: 132570:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4098.843309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.396325] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4108.444069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4108.456127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4108.644607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4115.368881] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4119.958295] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4120.147194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4124.084407] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4124.240866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4129.253154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.272030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4129.274967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4132.614470] Lustre: server umount lustre-OST0000 complete [ 4139.493848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.510440] Lustre: Skipped 1 previous similar message [ 4139.518154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.536693] Lustre: Skipped 2 previous similar messages [ 4143.650716] Lustre: server umount lustre-MDT0000 complete [ 4147.272839] LustreError: 130593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736444 with bad export cookie 4072331175301048404 [ 4147.281546] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4147.295170] LustreError: 130593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4147.939150] Lustre: server umount lustre-MDT0001 complete [ 4157.472690] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:40:54 (1786736454) [ 4164.087983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.956727] LustreError: 134983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.971847] LustreError: 134983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4165.035296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.645862] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4170.233219] LustreError: 134984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.248492] LustreError: 134984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4175.331138] LustreError: 134983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.597221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.105866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.304809] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4192.069733] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4198.728443] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4201.318035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.330134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.335981] Lustre: Skipped 2 previous similar messages [ 4201.363517] Lustre: Skipped 1 previous similar message [ 4207.598053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.611636] Lustre: Skipped 3 previous similar messages [ 4208.350389] Lustre: server umount lustre-MDT0000 complete [ 4212.714888] LustreError: 134979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.741830] LustreError: 134979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4213.214806] LustreError: 134964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736510 with bad export cookie 4072331175301049293 [ 4213.237879] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4214.066902] Lustre: server umount lustre-MDT0001 complete [ 4224.264988] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:42:00 (1786736520) [ 4231.427244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.021265] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4252.193059] LustreError: 137398:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 4277.920383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4284.817651] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4292.084569] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4293.912367] Lustre: server umount lustre-OST0000 complete [ 4301.919827] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:43:18 (1786736598) [ 4307.307679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.902618] LustreError: 138843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.990967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.853707] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4318.191731] LustreError: 138844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.226018] LustreError: 138844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4323.095142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.700262] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4336.108872] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4341.558227] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4346.756894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.993481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.996055] Lustre: Skipped 1 previous similar message [ 4354.281326] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4361.270260] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4369.891537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4369.912461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.930788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.990773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.700585] Lustre: server umount lustre-MDT0000 complete [ 4379.110772] LustreError: 141339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.144117] LustreError: 141339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4380.839033] LustreError: 139578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736677 with bad export cookie 4072331175301049951 [ 4380.858638] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4380.862934] LustreError: 139578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4381.566828] Lustre: server umount lustre-MDT0001 complete [ 4396.994831] Lustre: server umount lustre-OST0000 complete [ 4404.515717] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:45:01 (1786736701) [ 4411.996849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.486400] LustreError: 142502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.539368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.945504] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4424.082469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.097651] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4436.361720] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4441.400384] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4448.200708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.117103] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4456.974083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4461.262724] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4465.857198] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4466.064929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4470.197822] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4470.397161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4472.293516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.301835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4472.318725] Lustre: Skipped 1 previous similar message [ 4472.334546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4472.334554] Lustre: Skipped 3 previous similar messages [ 4486.112922] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4486.263470] Lustre: server umount lustre-OST0000 complete [ 4491.234866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4491.242078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.252612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.257517] Lustre: Skipped 5 previous similar messages [ 4495.838645] Lustre: server umount lustre-MDT0000 complete [ 4497.889148] LustreError: 142497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.896115] LustreError: 142497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4499.157909] LustreError: 142482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736796 with bad export cookie 4072331175301051022 [ 4499.161447] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4499.166649] LustreError: 142482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4499.471441] Lustre: server umount lustre-MDT0001 complete [ 4507.994143] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:46:44 (1786736804) [ 4514.582553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4530.080777] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4535.201875] LustreError: 146661:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 4560.928758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.946552] Lustre: Skipped 2 previous similar messages [ 4566.516772] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4573.594627] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4579.008096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.447363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4584.058984] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4592.102649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.905867] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4603.169474] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4609.398985] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4614.303730] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4614.564547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4621.080479] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4621.462071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4626.415444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.437300] Lustre: Skipped 1 previous similar message [ 4626.446711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4626.454774] Lustre: Skipped 4 previous similar messages [ 4631.250616] Lustre: server umount lustre-MDT0000 complete [ 4631.521905] LustreError: 146686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.538536] LustreError: 146686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4635.718323] LustreError: 146667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786736932 with bad export cookie 4072331175301052051 [ 4635.753088] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.310160] Lustre: server umount lustre-MDT0001 complete [ 4651.153097] Lustre: server umount lustre-OST0000 complete [ 4660.048313] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:49:16 (1786736956) [ 4666.269056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.016690] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4687.136650] LustreError: 150853:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 4719.609126] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4726.879622] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4732.996774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.253252] LustreError: 128824:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000cd1b0ce x1873528815247872/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4733.710791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4737.784401] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4746.097783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.381974] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4758.811648] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4764.976711] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4771.827107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4772.045475] Lustre: lustre-OST0001: new disk, initializing [ 4772.056351] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4772.066242] Lustre: 153728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4778.321758] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4780.133471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4780.148659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4780.249902] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4788.598884] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4792.361771] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4792.568902] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4796.667450] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4796.853245] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4800.482290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4800.505245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.545790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4804.894914] Lustre: server umount lustre-OST0000 complete [ 4811.240655] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4814.679224] Lustre: server umount lustre-OST0001 complete [ 4824.541279] Lustre: server umount lustre-MDT0000 complete [ 4828.300827] LustreError: 150859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786737125 with bad export cookie 4072331175301053003 [ 4828.303659] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.385756] LustreError: 150859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4828.856731] Lustre: server umount lustre-MDT0001 complete [ 4854.182504] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:52:30 (1786737150) [ 4855.499542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4857.009440] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4858.468688] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:52:35 (1786737155) [ 4864.507246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.817217] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4864.852128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4865.000878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.008991] Lustre: Skipped 6 previous similar messages [ 4870.281734] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4877.872061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.013175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4878.071687] Lustre: 157290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4881.695968] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4890.298642] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4898.071515] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4905.508111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4905.738793] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4911.575131] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4915.325194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4917.221538] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4920.844467] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4921.068918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4925.147046] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4925.390520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4930.530870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4930.547677] LustreError: Skipped 2 previous similar messages [ 4930.558292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.576317] Lustre: Skipped 5 previous similar messages [ 4930.581515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4930.591807] Lustre: Skipped 6 previous similar messages [ 4942.307279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4942.803801] Lustre: server umount lustre-OST0000 complete [ 4965.546367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4970.988937] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.998813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.000556] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4972.005480] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4979.052907] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4983.694074] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4983.968254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4988.260262] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4988.402742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4993.172877] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff998806f91800.ost_server_uuid 50 [ 4994.303259] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff998806f91800.ost_server_uuid in FULL state after 0 sec [ 4994.414784] LustreError: 157297:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.21@tcp) returned error from blocking AST (req@0000000011f837aa x1873528815352064 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a4e2040f/0x3883d3a0b79183a5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.21@tcp remote: 0x3be246262941082 expref: 5 pid: 157295 timeout: 5094 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4994.431254] LustreError: lustre-MDT0000: A client on nid 192.168.202.21@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4994.436031] LustreError: 157285:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.21@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a4e2040f/0x3883d3a0b79183a5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.21@tcp remote: 0x3be246262941082 expref: 6 pid: 157295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5000.178226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5000.185092] LustreError: Skipped 1 previous similar message [ 5004.319889] Lustre: server umount lustre-OST0000 complete [ 5014.666758] Lustre: server umount lustre-MDT0000 complete [ 5017.059325] LustreError: 157301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.081655] LustreError: 157301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5019.414231] LustreError: 157281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786737316 with bad export cookie 4072331175301054179 [ 5019.420502] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5019.430828] LustreError: 157281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5026.123946] Lustre: server umount lustre-MDT0001 complete [ 5037.122361] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5040.188379] Key type lgssc unregistered [ 5040.567778] LNet: 163404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5040.572385] LNetError: 163404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5040.592576] LNet: Removed LNI 192.168.202.121@tcp [ 5041.390450] Key type .llcrypt unregistered [ 5041.393255] Key type ._llcrypt unregistered [ 5065.855246] Key type ._llcrypt registered [ 5065.858187] Key type .llcrypt registered [ 5065.971805] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5067.299210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5067.491492] alg: No test for adler32 (adler32-zlib) [ 5068.705610] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 5069.036241] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5070.760235] Key type lgssc registered [ 5071.796790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5081.379813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5081.396918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.896745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.236649] LustreError: 164819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.273122] LustreError: 164819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5088.305377] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5092.325489] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.273853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.456887] LustreError: 164819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.829245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5103.643535] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5112.022495] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5119.965795] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5127.854356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5128.377919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.514904] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5142.433395] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5151.918829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5159.212094] Lustre: Failing over lustre-MDT0000 [ 5159.329761] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 5159.397567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5159.405235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.418666] LustreError: 164819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.558715] Lustre: server umount lustre-MDT0000 complete [ 5162.473457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5162.483196] Lustre: Skipped 1 previous similar message [ 5167.590477] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.615470] LustreError: 164820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5171.391028] LustreError: 164799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.21@tcp arrived at 1786737468 with bad export cookie 5877109490893094592 [ 5177.829350] LustreError: 167311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.845873] LustreError: 167311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5178.202196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.393610] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5178.550584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.588834] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5182.608193] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5183.982805] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5183.991331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5184.839917] Lustre: lustre-MDT0000: Denying connection for new client d169798f-e2c9-4868-b21d-9c99e81360de (at 192.168.202.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5190.010481] Lustre: lustre-MDT0000: Denying connection for new client d169798f-e2c9-4868-b21d-9c99e81360de (at 192.168.202.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5194.217410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.217871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5194.237403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.332719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.337825] Lustre: Skipped 2 previous similar messages [ 5200.173528] Lustre: server umount lustre-OST0000 complete [ 5203.628403] LustreError: 168585:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5203.646544] Lustre: 167934:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5203.654653] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5203.668763] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5203.678619] LustreError: 167934:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e659d6c4 x1873529920338304/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5203.691764] LustreError: 167934:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5203.700237] LustreError: 167934:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5203.710766] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5204.107597] Lustre: server umount lustre-MDT0000 complete [ 5204.459719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.482414] Lustre: Skipped 1 previous similar message [ 5204.485425] LustreError: 167221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.683764] LustreError: 164800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786737505 with bad export cookie 5877109490893094921 [ 5208.691042] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5208.693884] LustreError: 164800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5215.052570] Lustre: server umount lustre-MDT0001 complete [ 5227.746849] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5230.709541] Key type lgssc unregistered [ 5231.143681] LNet: 169339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5231.150804] LNetError: 169339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5231.170663] LNet: Removed LNI 192.168.202.121@tcp [ 5231.854082] Key type .llcrypt unregistered [ 5231.857758] Key type ._llcrypt unregistered [ 5255.736596] Key type ._llcrypt registered [ 5255.738681] Key type .llcrypt registered [ 5255.888499] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5257.692573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5257.744801] alg: No test for adler32 (adler32-zlib) [ 5258.917504] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 5259.220901] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5260.912626] Key type lgssc registered [ 5262.403808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5269.864143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5269.888474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.303820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5275.155641] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5281.763468] LustreError: 170751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.777148] LustreError: 170751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5283.723836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.308423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5288.621903] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5297.134436] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5302.452665] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5309.809795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.357675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.557105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5319.167462] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5327.302184] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5335.009143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5335.012791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.017498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5335.029650] Lustre: Skipped 1 previous similar message [ 5335.039552] Lustre: Skipped 1 previous similar message [ 5340.081951] Lustre: server umount lustre-OST0000 complete [ 5345.250624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.252245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5345.264450] Lustre: Skipped 1 previous similar message [ 5345.277212] Lustre: Skipped 1 previous similar message [ 5350.267367] Lustre: server umount lustre-MDT0000 complete [ 5350.372767] LustreError: 170746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5350.392681] LustreError: 170746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5353.847353] LustreError: 170732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786737650 with bad export cookie 18202341334308106984 [ 5353.852541] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5353.860084] LustreError: 170732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5354.244537] Lustre: server umount lustre-MDT0001 complete [ 5365.777940] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5368.338624] Key type lgssc unregistered [ 5368.601524] LNet: 174295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.608280] LNetError: 174295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.623800] LNet: Removed LNI 192.168.202.121@tcp [ 5369.489131] Key type .llcrypt unregistered [ 5369.491736] Key type ._llcrypt unregistered [ 5403.272308] Key type ._llcrypt registered [ 5403.275526] Key type .llcrypt registered [ 5403.466686] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5404.435668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5404.550665] alg: No test for adler32 (adler32-zlib) [ 5405.832783] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 5406.165625] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5407.944245] Key type lgssc registered [ 5409.417856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5417.732236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5417.750427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.261704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.706530] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5429.734417] LustreError: 175998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.753206] LustreError: 175998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5433.442190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.058141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5437.959579] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5444.907645] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5449.789299] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5456.156366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.463121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.687646] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5464.689033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5470.864887] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5480.051305] Lustre: server umount lustre-OST0000 complete [ 5485.026677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5485.031269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.039050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5485.542340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.551930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.658817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5494.072080] Lustre: server umount lustre-MDT0000 complete [ 5495.790800] LustreError: 175999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.818969] LustreError: 175999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5496.800246] Lustre: 175309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786737777/real 1786737777] req@000000000d1dbf50 x1873530273685504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786737793 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5496.843748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.872785] LustreError: 175998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.887884] LustreError: 175998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5498.331075] LustreError: 175978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786737795 with bad export cookie 16759997324408302996 [ 5498.334358] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5498.349594] LustreError: 175978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5498.643530] Lustre: server umount lustre-MDT0001 complete [ 5509.445202] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5512.726127] Key type lgssc unregistered [ 5512.936800] LNet: 179530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5512.940670] LNetError: 179530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5512.950685] LNet: Removed LNI 192.168.202.121@tcp [ 5513.603766] Key type .llcrypt unregistered [ 5513.605628] Key type ._llcrypt unregistered [ 5538.913313] Key type ._llcrypt registered [ 5538.915778] Key type .llcrypt registered [ 5539.098486] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5541.099582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5541.146816] alg: No test for adler32 (adler32-zlib) [ 5542.576750] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 5542.933941] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5544.670986] Key type lgssc registered [ 5546.114989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5558.029682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5558.050257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.761131] Lustre: *** cfs_fail_loc=135, val=0*** [ 5559.917758] LustreError: 181055:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5559.925270] LustreError: 181055:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.121@tcp: cfg command failed: rc = -2 [ 5559.933714] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5559.933714] [ 5559.948311] LustreError: MGC192.168.202.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5559.977639] LustreError: 181016:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5559.989952] LustreError: 181016:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5560.012053] LustreError: 181016:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5560.018136] LustreError: 181016:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5560.136305] Lustre: server umount lustre-MDT0000 complete [ 5560.141221] LustreError: 181016:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5567.853048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.193781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.048943] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5578.724875] LustreError: 181427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.738461] LustreError: 181427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5578.948615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.248944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5582.979635] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5589.158572] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5594.729194] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5598.689137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.717209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.721389] Lustre: Skipped 1 previous similar message [ 5602.870384] Lustre: server umount lustre-MDT0000 complete [ 5603.818933] LustreError: 181427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5603.841421] LustreError: 181427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5604.836475] LustreError: 181428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.852431] LustreError: 181428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5606.226929] LustreError: 181407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786737903 with bad export cookie 11400666150028343985 [ 5606.230702] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.233806] LustreError: 181407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5606.517789] Lustre: server umount lustre-MDT0001 complete [ 5615.286374] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5617.844610] Key type lgssc unregistered [ 5618.068754] LNet: 183745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5618.072977] LNetError: 183745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5619.111688] LNet: Removed LNI 192.168.202.121@tcp [ 5619.781499] Key type .llcrypt unregistered [ 5619.788376] Key type ._llcrypt unregistered [ 5639.397553] Key type ._llcrypt registered [ 5639.399601] Key type .llcrypt registered [ 5639.490625] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5639.730435] Key type .llcrypt unregistered [ 5639.732888] Key type ._llcrypt unregistered [ 5655.781956] Key type ._llcrypt registered [ 5655.790843] Key type .llcrypt registered [ 5655.948650] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5657.335161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5657.377836] alg: No test for adler32 (adler32-zlib) [ 5658.400282] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 5658.618095] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5660.264200] Key type lgssc registered [ 5661.362492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5668.026658] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5668.043687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.552964] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5688.672278] LustreError: 185870:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 5715.424190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5720.879934] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5728.269983] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5734.976606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.456062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.516365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5740.261735] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5740.525700] LustreError: 185895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.545810] LustreError: 185895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5745.635571] LustreError: 185896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.266934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.670705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5754.701925] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5761.918581] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5767.955107] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5770.915264] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5780.450144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.470154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.490291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5781.478734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.484583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5781.505284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5785.678740] Lustre: server umount lustre-OST0000 complete [ 5790.695569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5790.703814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.731652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5790.737937] Lustre: Skipped 1 previous similar message [ 5796.337920] Lustre: server umount lustre-MDT0000 complete [ 5796.834371] LustreError: 186912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.852882] LustreError: 186912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5800.045766] LustreError: 188534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786738097 with bad export cookie 15087705509022412857 [ 5800.058681] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5800.290793] Lustre: server umount lustre-MDT0001 complete [ 5810.861956] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5813.718676] Key type lgssc unregistered [ 5814.041378] LNet: 189788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5814.048452] LNetError: 189788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5815.081103] LNet: Removed LNI 192.168.202.121@tcp [ 5815.967974] Key type .llcrypt unregistered [ 5815.970145] Key type ._llcrypt unregistered [ 5838.700923] Key type ._llcrypt registered [ 5838.706386] Key type .llcrypt registered [ 5838.812810] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5840.267897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5840.319869] alg: No test for adler32 (adler32-zlib) [ 5841.366323] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 5841.689651] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5843.436229] Key type lgssc registered [ 5845.090118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5852.666403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5852.679656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.257741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.065501] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5864.460734] LustreError: 191200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.481914] LustreError: 191200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5869.547517] LustreError: 191201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.215785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.576433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5875.662269] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5883.743670] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5892.594556] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5900.355340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.104241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.552930] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5919.003413] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.717312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5927.795657] Lustre: Failing over lustre-MDT0000 [ 5927.900797] LustreError: 193833:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000008e7ec05c) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5928.100549] Lustre: server umount lustre-MDT0000 complete [ 5929.879807] LustreError: 191197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.959033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.984226] Lustre: Skipped 1 previous similar message [ 5934.963673] LustreError: 191195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.985910] LustreError: 191195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5940.102833] LustreError: 191196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.142628] LustreError: 191196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5946.340680] Lustre: 190512:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786738227/real 1786738227] req@00000000aa62bb79 x1873530729819776/t0(0) o400->MGC192.168.202.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786738243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5946.381477] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.619552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.321505] LustreError: 191197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.347458] LustreError: 191197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5956.619970] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x81edabbab556befd [ 5956.645542] Lustre: MGC192.168.202.121@tcp: Connection restored to 0@lo (at 0@lo) [ 5957.000671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.100874] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5960.566832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5961.499477] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5962.232450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5962.283356] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5962.338176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5967.379989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5973.971683] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5982.183109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5982.209072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.216215] Lustre: Skipped 1 previous similar message [ 5982.220919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.035879] Lustre: server umount lustre-OST0000 complete [ 5992.929776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5992.932867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.938607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.938616] Lustre: Skipped 1 previous similar message [ 5992.999588] Lustre: Skipped 2 previous similar messages [ 5996.615401] Lustre: server umount lustre-MDT0000 complete [ 5998.060637] LustreError: 193696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.081973] LustreError: 193696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6000.435386] LustreError: 191181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786738297 with bad export cookie 9362328018767757053 [ 6000.439575] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6000.444680] LustreError: 191181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6000.728232] Lustre: server umount lustre-MDT0001 complete [ 6013.194929] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6017.146281] Key type lgssc unregistered [ 6017.613806] LNet: 196044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6017.617821] LNetError: 196044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6017.634939] LNet: Removed LNI 192.168.202.121@tcp [ 6018.932278] Key type .llcrypt unregistered [ 6018.936960] Key type ._llcrypt unregistered [ 6044.562824] Key type ._llcrypt registered [ 6044.564853] Key type .llcrypt registered [ 6044.685903] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6046.132849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6046.181865] alg: No test for adler32 (adler32-zlib) [ 6047.469719] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 6047.955895] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6049.666622] Key type lgssc registered [ 6050.826986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6060.079658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6060.097861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.730452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.966083] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6071.267434] LustreError: 197457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.288460] LustreError: 197457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6074.825528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.170197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.647477] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6087.309775] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6095.118603] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.760234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6102.212232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.291036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6108.045367] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6116.289824] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6120.888390] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6124.846340] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6141.569183] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6147.555062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.555701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.573812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.600588] Lustre: Skipped 1 previous similar message [ 6152.648313] Lustre: server umount lustre-OST0000 complete [ 6157.284703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6157.298506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.316657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.319759] Lustre: Skipped 1 previous similar message [ 6162.914745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.923723] Lustre: Skipped 1 previous similar message [ 6163.000491] Lustre: server umount lustre-MDT0000 complete [ 6167.012717] LustreError: 197458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.040786] LustreError: 197458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6167.573549] LustreError: 199991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786738464 with bad export cookie 10213556141968040222 [ 6167.585204] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6167.588618] LustreError: 199991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6167.903766] Lustre: server umount lustre-MDT0001 complete [ 6178.663377] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6181.495928] Key type lgssc unregistered [ 6181.778145] LNet: 201322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6181.791328] LNetError: 201322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6181.813844] LNet: Removed LNI 192.168.202.121@tcp [ 6182.704461] Key type .llcrypt unregistered [ 6182.706058] Key type ._llcrypt unregistered [ 6202.801669] Key type ._llcrypt registered [ 6202.808897] Key type .llcrypt registered [ 6202.894770] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6204.051365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6204.244122] alg: No test for adler32 (adler32-zlib) [ 6205.239984] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 6205.455901] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6207.113443] Key type lgssc registered [ 6208.147852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6219.358116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6219.382483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.917299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6223.928099] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6231.525766] LustreError: 202592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6231.548075] LustreError: 202592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6234.643502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.373574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.149352] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6246.668358] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6251.326872] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6258.338813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6258.676059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.240419] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6266.960344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6272.501189] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6278.276647] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6288.010460] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6299.556226] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6307.273528] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6322.160545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6322.173487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.190515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6323.172136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6323.175960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6323.190410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.221745] Lustre: Skipped 1 previous similar message [ 6324.684547] Lustre: server umount lustre-OST0000 complete [ 6332.387028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6332.401092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.433777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.549944] Lustre: server umount lustre-MDT0000 complete [ 6338.529597] LustreError: 202592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.564063] LustreError: 202592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6339.784027] LustreError: 205269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786738636 with bad export cookie 9576779176636624887 [ 6339.796318] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6339.797867] LustreError: 205269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6340.113486] Lustre: server umount lustre-MDT0001 complete [ 6351.586413] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6354.731174] Key type lgssc unregistered [ 6355.001640] LNet: 207563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6355.005714] LNetError: 207563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6356.073775] LNet: Removed LNI 192.168.202.121@tcp [ 6356.876423] Key type .llcrypt unregistered [ 6356.878260] Key type ._llcrypt unregistered [ 6380.791743] Key type ._llcrypt registered [ 6380.799506] Key type .llcrypt registered [ 6380.951428] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6382.119448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6382.202847] alg: No test for adler32 (adler32-zlib) [ 6383.282456] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 6383.585404] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6385.289876] Key type lgssc registered [ 6386.535173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6393.073881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6393.094272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.487166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.615938] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6404.068418] LustreError: 208978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6404.101036] LustreError: 208978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6407.438457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6407.835372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6412.650547] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6420.085172] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6426.042736] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6432.330801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6432.780539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.458664] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6439.916635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6446.396743] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.388186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6460.390643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.392936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6460.408357] Lustre: Skipped 1 previous similar message [ 6465.188972] Lustre: server umount lustre-OST0000 complete [ 6470.625616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.629261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6470.641765] Lustre: Skipped 1 previous similar message [ 6470.660750] Lustre: Skipped 2 previous similar messages [ 6475.745209] Lustre: server umount lustre-MDT0000 complete [ 6475.748499] LustreError: 209706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.769767] LustreError: 209706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6479.369830] LustreError: 208958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786738776 with bad export cookie 15317483441073659405 [ 6479.376872] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.381621] LustreError: 208958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6479.866265] Lustre: server umount lustre-MDT0001 complete [ 6491.039191] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6493.738573] Key type lgssc unregistered [ 6494.086800] LNet: 212659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6494.099138] LNetError: 212659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6494.124358] LNet: Removed LNI 192.168.202.121@tcp [ 6495.088185] Key type .llcrypt unregistered [ 6495.091090] Key type ._llcrypt unregistered [ 6515.413282] Key type ._llcrypt registered [ 6515.416377] Key type .llcrypt registered [ 6515.553905] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6517.112520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6517.369077] alg: No test for adler32 (adler32-zlib) [ 6518.519168] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 6518.840354] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6520.545531] Key type lgssc registered [ 6521.562536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6530.587890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6530.602993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.205227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6535.703227] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6542.831373] LustreError: 213927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.849179] LustreError: 213927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6544.205785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.505095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6548.406630] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6556.076814] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6562.844137] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6570.550368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.049501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6577.214996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6577.375603] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6585.295781] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6597.602331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6597.611749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.632900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6599.664434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6599.665315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6599.676298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.732155] Lustre: Skipped 1 previous similar message [ 6601.799067] Lustre: server umount lustre-OST0000 complete [ 6609.890428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.893466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6609.916251] Lustre: Skipped 1 previous similar message [ 6609.936096] Lustre: Skipped 1 previous similar message [ 6612.451267] Lustre: server umount lustre-MDT0000 complete [ 6615.010796] LustreError: 215704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.041765] LustreError: 215704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6617.384964] LustreError: 213908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786738914 with bad export cookie 2946133275093971386 [ 6617.413289] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.880762] Lustre: server umount lustre-MDT0001 complete [ 6631.891607] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6637.115040] Key type lgssc unregistered [ 6637.749646] LNet: 217610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6637.763314] LNetError: 217610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.789145] LNet: Removed LNI 192.168.202.121@tcp [ 6639.517430] Key type .llcrypt unregistered [ 6639.518862] Key type ._llcrypt unregistered [ 6662.405350] Key type ._llcrypt registered [ 6662.412078] Key type .llcrypt registered [ 6662.605622] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6663.896216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6663.921112] alg: No test for adler32 (adler32-zlib) [ 6664.943433] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 6665.232840] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6666.868613] Key type lgssc registered [ 6668.181210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6676.804842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6676.838749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.561342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.589456] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6688.737271] LustreError: 218875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6688.757667] LustreError: 218875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6693.857700] LustreError: 218876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.740211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.197429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.106828] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6707.684713] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6711.772341] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.888362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6719.197950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6723.321185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6724.799111] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6732.235360] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6741.199872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6741.273739] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6741.557441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6750.181481] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6757.962624] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6759.546343] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6771.349845] Lustre: Permanently deactivating lustre-OST0001 [ 6771.365309] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6781.486626] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6787.087840] Lustre: Permanently reactivating lustre-OST0001 [ 6787.097338] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6787.100813] Lustre: Skipped 2 previous similar messages [ 6796.833665] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.842563] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6800.970806] Lustre: server umount lustre-OST0001 complete [ 6801.913454] LustreError: 220698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.473552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6805.488568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.520932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.016852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.017408] LustreError: 222076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.030102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.956324] Lustre: server umount lustre-OST0000 complete [ 6817.248239] LustreError: 222782:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 6817.272387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.282221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.301811] Lustre: Skipped 1 previous similar message [ 6817.323892] Lustre: Skipped 2 previous similar messages [ 6822.369975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6822.387604] Lustre: Skipped 1 previous similar message [ 6824.512326] Lustre: server umount lustre-MDT0000 complete [ 6827.499511] LustreError: 218876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6827.531327] LustreError: 218876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6827.902142] LustreError: 218854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786739125 with bad export cookie 16858719089328114283 [ 6827.904297] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6827.913964] LustreError: 218854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6828.213565] Lustre: server umount lustre-MDT0001 complete [ 6837.827496] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6839.974621] Key type lgssc unregistered [ 6840.202885] LNet: 223937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6840.212452] LNetError: 223937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6840.237638] LNet: Removed LNI 192.168.202.121@tcp [ 6841.157462] Key type .llcrypt unregistered [ 6841.160314] Key type ._llcrypt unregistered [ 6882.596737] Key type ._llcrypt registered [ 6882.604741] Key type .llcrypt registered [ 6882.788808] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6885.068370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6885.182763] alg: No test for adler32 (adler32-zlib) [ 6886.782674] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 6887.236761] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6889.112193] Key type lgssc registered [ 6891.446878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6900.138262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6900.156296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.514491] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6901.549521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6901.763739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.380798] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6916.669838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.799669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6916.879969] Lustre: 226145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6917.307203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6921.810188] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6929.283734] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6935.212616] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6941.399399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.584121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6941.765726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.191592] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6948.491910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6954.495258] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6958.273081] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6965.312925] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6973.909556] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6983.220728] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6996.204616] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6996.217432] Lustre: Skipped 1 previous similar message [ 7020.131436] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7020.143092] Lustre: Skipped 2 previous similar messages [ 7053.670733] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7053.680935] Lustre: Skipped 4 previous similar messages [ 7121.365545] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7121.371471] Lustre: Skipped 7 previous similar messages [ 7127.522412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7127.527614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.537414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7128.034231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7128.042213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.073928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.115649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.124678] Lustre: Skipped 1 previous similar message [ 7133.551715] Lustre: server umount lustre-OST0000 complete [ 7141.352658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.371610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.376627] Lustre: Skipped 1 previous similar message [ 7141.409740] Lustre: Skipped 1 previous similar message [ 7144.291145] Lustre: server umount lustre-MDT0000 complete [ 7149.014273] LustreError: 228713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786739446 with bad export cookie 18257134511499668251 [ 7149.015484] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7149.021081] LustreError: 228713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7149.431263] Lustre: server umount lustre-MDT0001 complete [ 7160.434625] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7163.182091] Key type lgssc unregistered [ 7163.497950] LNet: 230943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7163.506538] LNetError: 230943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7163.540365] LNet: Removed LNI 192.168.202.121@tcp [ 7164.556272] Key type .llcrypt unregistered [ 7164.558505] Key type ._llcrypt unregistered [ 7188.559384] Key type ._llcrypt registered [ 7188.562406] Key type .llcrypt registered [ 7188.686790] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7190.567778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7190.656254] alg: No test for adler32 (adler32-zlib) [ 7191.931358] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 7192.287457] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7194.072267] Key type lgssc registered [ 7195.743831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7204.155071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7204.185449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.637873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.346470] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7215.079236] LustreError: 232374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.109513] LustreError: 232374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7219.294442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.749965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7224.323289] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7232.256939] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.987642] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7244.310555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.594612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.738543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7251.266874] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7257.583212] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7274.471638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.482064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.493284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.519345] Lustre: Skipped 1 previous similar message [ 7274.530677] Lustre: Skipped 1 previous similar message [ 7279.669584] Lustre: server umount lustre-OST0000 complete [ 7284.711440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.721171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7284.725395] Lustre: Skipped 1 previous similar message [ 7289.803710] Lustre: server umount lustre-MDT0000 complete [ 7289.837660] LustreError: 232375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.862944] LustreError: 232375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7293.923621] LustreError: 233102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.944334] LustreError: 233102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7294.711771] LustreError: 232354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786739591 with bad export cookie 1703331041623504625 [ 7294.720478] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.730919] LustreError: 232354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7295.356537] Lustre: server umount lustre-MDT0001 complete [ 7307.558447] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7310.841902] Key type lgssc unregistered [ 7311.087704] LNet: 236063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7311.090872] LNetError: 236063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7311.103565] LNet: Removed LNI 192.168.202.121@tcp [ 7311.753605] Key type .llcrypt unregistered [ 7311.756417] Key type ._llcrypt unregistered [ 7334.651921] Key type ._llcrypt registered [ 7334.654265] Key type .llcrypt registered [ 7334.768920] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7335.967289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7336.039091] alg: No test for adler32 (adler32-zlib) [ 7337.024622] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 7337.319072] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7338.992231] Key type lgssc registered [ 7340.120929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7611.510726] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7614.341961] Key type lgssc unregistered [ 7614.797150] LNet: 238088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7614.815988] LNetError: 238088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7614.855218] LNet: Removed LNI 192.168.202.121@tcp [ 7615.654415] Key type .llcrypt unregistered [ 7615.656349] Key type ._llcrypt unregistered [ 7640.816645] Key type ._llcrypt registered [ 7640.822183] Key type .llcrypt registered [ 7640.917672] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7656.371466] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7657.493571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.513898] alg: No test for adler32 (adler32-zlib) [ 7658.638458] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 7658.869888] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7660.536184] Key type lgssc registered [ 7662.223456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7696.183859] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7396 sec ========== 16:39:52 (1786739992) [ 7697.534924] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:39:54 (1786739994) === [ 7700.169515] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:39:57 (1786739997) === [ 7737.842083] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7740.589589] Key type lgssc unregistered [ 7740.931627] LNet: 243325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7740.937819] LNetError: 243325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7740.953302] LNet: Removed LNI 192.168.202.121@tcp [ 7741.914461] Key type .llcrypt unregistered [ 7741.916760] Key type ._llcrypt unregistered