[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 447968860 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002286] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007529] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.020471] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025699] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.030398] *** VALIDATE proc *** [ 0.031218] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034175] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036099] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041764] debug: unmapping init [mem 0xffffffff9e459000-0xffffffff9e460fff] [ 0.043164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052302] rcu: Hierarchical SRCU implementation. [ 0.054268] smp: Bringing up secondary CPUs ... [ 0.055553] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.059461] smp: Brought up 1 node, 4 CPUs [ 0.061008] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144169] node 0 deferred pages initialised in 80ms [ 0.147088] devtmpfs: initialized [ 0.148226] x86/mm: Memory block size: 128MB [ 0.150582] gcov: version magic: 0x41383552 [ 0.153000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156240] pinctrl core: initialized pinctrl subsystem [ 0.158210] [ 0.158564] ************************************************************* [ 0.160009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161009] ** ** [ 0.163009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164008] ** ** [ 0.166009] ** This means that this kernel is built to expose internal ** [ 0.167007] ** IOMMU data structures, which may compromise security on ** [ 0.168008] ** your system. ** [ 0.170009] ** ** [ 0.171007] ** If you see this message and you are not debugging the ** [ 0.173009] ** kernel, report this immediately to your vendor! ** [ 0.175009] ** ** [ 0.176007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178009] ************************************************************* [ 0.179635] NET: Registered protocol family 16 [ 0.181441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190012] cpuidle: using governor menu [ 0.191530] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194634] PCI: Using configuration type 1 for base access [ 0.196123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205096] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212048] cryptd: max_cpu_qlen set to 1000 [ 0.214309] ACPI: Added _OSI(Module Device) [ 0.216016] ACPI: Added _OSI(Processor Device) [ 0.218006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224245] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229492] ACPI: Interpreter enabled [ 0.231040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232011] ACPI: Using IOAPIC for interrupt routing [ 0.234125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246237] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258337] acpiphp: Slot [2] registered [ 0.260104] acpiphp: Slot [5] registered [ 0.261110] acpiphp: Slot [6] registered [ 0.263091] acpiphp: Slot [7] registered [ 0.264095] acpiphp: Slot [8] registered [ 0.265074] acpiphp: Slot [9] registered [ 0.266104] acpiphp: Slot [10] registered [ 0.268107] acpiphp: Slot [3] registered [ 0.269086] acpiphp: Slot [4] registered [ 0.270097] acpiphp: Slot [11] registered [ 0.272176] acpiphp: Slot [12] registered [ 0.274105] acpiphp: Slot [13] registered [ 0.275124] acpiphp: Slot [14] registered [ 0.277133] acpiphp: Slot [15] registered [ 0.279107] acpiphp: Slot [16] registered [ 0.280102] acpiphp: Slot [17] registered [ 0.282103] acpiphp: Slot [18] registered [ 0.284086] acpiphp: Slot [19] registered [ 0.286112] acpiphp: Slot [20] registered [ 0.287088] acpiphp: Slot [21] registered [ 0.289086] acpiphp: Slot [22] registered [ 0.290107] acpiphp: Slot [23] registered [ 0.292137] acpiphp: Slot [24] registered [ 0.293092] acpiphp: Slot [25] registered [ 0.295097] acpiphp: Slot [26] registered [ 0.296073] acpiphp: Slot [27] registered [ 0.298080] acpiphp: Slot [28] registered [ 0.299073] acpiphp: Slot [29] registered [ 0.300081] acpiphp: Slot [30] registered [ 0.301072] acpiphp: Slot [31] registered [ 0.302046] PCI host bridge to bus 0000:00 [ 0.303015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325293] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334783] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340560] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351316] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362216] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.378634] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.384015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392640] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.404017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.427098] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.471788] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.476012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.480014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.501727] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.507012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.512015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.533101] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.540013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.546015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.563015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.573291] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.578013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.582200] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.599013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.609195] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.611366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.614353] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.616338] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.619264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.626040] iommu: Default domain type: Passthrough [ 0.628393] SCSI subsystem initialized [ 0.629167] ACPI: bus type USB registered [ 0.631120] usbcore: registered new interface driver usbfs [ 0.633078] usbcore: registered new interface driver hub [ 0.635056] usbcore: registered new device driver usb [ 0.636107] pps_core: LinuxPPS API ver. 1 registered [ 0.637008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.639055] PTP clock support registered [ 0.641119] EDAC MC: Ver: 3.0.0 [ 0.644130] PCI: Using ACPI for IRQ routing [ 0.645677] NetLabel: Initializing [ 0.647010] NetLabel: domain hash size = 128 [ 0.648007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.649086] NetLabel: unlabeled traffic allowed by default [ 0.651119] vgaarb: loaded [ 0.652317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.655015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.661480] clocksource: Switched to clocksource kvm-clock [ 0.774231] VFS: Disk quotas dquot_6.6.0 [ 0.776154] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.779048] *** VALIDATE ramfs *** [ 0.780345] *** VALIDATE hugetlbfs *** [ 0.782242] pnp: PnP ACPI init [ 0.784697] pnp: PnP ACPI: found 6 devices [ 0.806863] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810322] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812213] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814229] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.816330] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.818089] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.820239] NET: Registered protocol family 2 [ 0.822178] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.827418] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.830823] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.836042] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.839660] TCP: Hash tables configured (established 65536 bind 65536) [ 0.842876] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.846147] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849570] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852835] NET: Registered protocol family 1 [ 0.855550] RPC: Registered named UNIX socket transport module. [ 0.857819] RPC: Registered udp transport module. [ 0.859215] RPC: Registered tcp transport module. [ 0.860766] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.862740] NET: Registered protocol family 44 [ 0.864972] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.867774] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.869855] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.872656] PCI: CLS 0 bytes, default 64 [ 0.874642] Unpacking initramfs... [ 2.278511] debug: unmapping init [mem 0xffff8f923cc54000-0xffff8f923ffbffff] [ 2.282905] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285526] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.288600] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.769394] Initialise system trusted keyrings [ 2.770913] Key type blacklist registered [ 2.772959] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.781269] zbud: loaded [ 2.784353] *** VALIDATE nfs *** [ 2.785313] *** VALIDATE nfs4 *** [ 2.786656] pstore: using deflate compression [ 2.789478] Platform Keyring initialized [ 2.885897] NET: Registered protocol family 38 [ 2.887207] Key type asymmetric registered [ 2.888259] Asymmetric key parser 'x509' registered [ 2.890023] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.894130] io scheduler mq-deadline registered [ 2.896126] io scheduler kyber registered [ 2.897965] io scheduler bfq registered [ 2.899789] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.902609] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.905353] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.908284] ACPI: Power Button [PWRF] [ 2.913869] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.920256] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.931183] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.937739] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.950097] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.980595] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.011311] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.016575] Non-volatile memory driver v1.3 [ 3.018272] Linux agpgart interface v0.103 [ 3.048956] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.052065] vda: detected capacity change from 0 to 74694656 [ 3.066297] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.068253] vdb: detected capacity change from 0 to 1073741824 [ 3.081324] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.083765] vdc: detected capacity change from 0 to 2621440000 [ 3.095968] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.098118] vdd: detected capacity change from 0 to 2621440000 [ 3.109236] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.111705] vde: detected capacity change from 0 to 4294967296 [ 3.125714] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.128247] vdf: detected capacity change from 0 to 4294967296 [ 3.135469] libphy: Fixed MDIO Bus: probed [ 3.140307] usbcore: registered new interface driver usbserial_generic [ 3.143872] usbserial: USB Serial support registered for generic [ 3.146432] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.150597] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.152273] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.154342] mousedev: PS/2 mouse device common for all mice [ 3.156732] rtc_cmos 00:05: RTC can wake from S4 [ 3.159161] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.160898] rtc_cmos 00:05: registered as rtc0 [ 3.164922] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.166613] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.168833] intel_pstate: CPU model not supported [ 3.174929] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.175238] hid: raw HID events driver (C) Jiri Kosina [ 3.180544] usbcore: registered new interface driver usbhid [ 3.182973] usbhid: USB HID core driver [ 3.185024] drop_monitor: Initializing network drop monitor service [ 3.187854] Initializing XFRM netlink socket [ 3.189651] NET: Registered protocol family 10 [ 3.192743] Segment Routing with IPv6 [ 3.194117] NET: Registered protocol family 17 [ 3.196017] mpls_gso: MPLS GSO support [ 3.203026] RAS: Correctable Errors collector initialized. [ 3.204792] AVX version of gcm_enc/dec engaged. [ 3.206082] AES CTR mode by8 optimization enabled [ 3.272853] sched_clock: Marking stable (3272824382, 0)->(4171131089, -898306707) [ 3.276224] registered taskstats version 1 [ 3.278168] Loading compiled-in X.509 certificates [ 3.279873] zswap: loaded using pool lzo/zbud [ 3.303046] Key type big_key registered [ 3.314804] Key type encrypted registered [ 3.316177] ima: No TPM chip found, activating TPM-bypass! [ 3.318055] ima: Allocated hash algorithm: sha1 [ 3.319540] ima: No architecture policies found [ 3.321315] evm: Initialising EVM extended attributes: [ 3.323511] evm: security.selinux [ 3.324526] evm: security.ima [ 3.325598] evm: security.capability [ 3.326889] evm: HMAC attrs: 0x1 [ 3.329646] rtc_cmos 00:05: setting system clock to 2026-08-15 13:28:58 UTC (1786800538) [ 3.337756] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.341269] debug: unmapping init [mem 0xffffffff9e182000-0xffffffff9e458fff] [ 3.349471] Write protecting the kernel read-only data: 28672k [ 3.353244] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 3.356658] debug: unmapping init [mem 0xffffffff9d114000-0xffffffff9d1fffff] [ 3.389804] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.396401] systemd[1]: Detected virtualization kvm. [ 3.398065] systemd[1]: Detected architecture x86-64. [ 3.399925] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.425549] systemd[1]: No hostname configured. [ 3.427399] systemd[1]: Set hostname to . [ 3.429134] random: systemd: uninitialized urandom read (16 bytes read) [ 3.431176] systemd[1]: Initializing machine ID from random generator. [ 3.468739] random: ln: uninitialized urandom read (6 bytes read) [ 3.545821] random: systemd: uninitialized urandom read (16 bytes read) [ 3.548234] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.552769] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.558031] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.219369] device-mapper: uevent: version 1.0.3 [ 4.221527] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.620381] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.980728] virtio_net virtio0 ens2: renamed from eth0 [ 5.115419] scsi host0: ata_piix [ 5.126218] scsi host1: ata_piix [ 5.130049] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.132640] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.886230] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.832834] random: crng init done [ 9.834276] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.351695] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.539591] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.779450] SELinux: Disabled at runtime. [ 11.841993] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.850781] systemd[1]: Detected virtualization kvm. [ 11.852772] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.402772] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.406824] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.412166] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.416101] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.420203] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.428905] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.433855] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.519121] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.974178] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.317487] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.332243] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.429286] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.443631] EDAC sbridge: Ver: 1.1.2 [ 16.777824] Key type dns_resolver registered [ 17.287449] NFS: Registering the id_resolver key type [ 17.291695] Key type id_resolver registered [ 17.295548] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg341-server login: [ 52.347844] hrtimer: interrupt took 3900289 ns [ 71.732245] libcfs: loading out-of-tree module taints kernel. [ 71.801567] Key type ._llcrypt registered [ 71.804327] Key type .llcrypt registered [ 71.903081] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 89.384869] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 90.798713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.809981] alg: No test for adler32 (adler32-zlib) [ 92.333996] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 93.212045] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 94.951186] Key type lgssc registered [ 96.566974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.961387] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.775594] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 178.329141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.352635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.603329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.640357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.772341] Lustre: lustre-MDT0000: new disk, initializing [ 179.816607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.826140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.354919] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.697788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.824986] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.870670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.882560] Lustre: Skipped 1 previous similar message [ 197.023674] Lustre: lustre-MDT0001: new disk, initializing [ 197.090204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.124641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 197.137312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.912692] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.470314] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.600379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.769182] Lustre: lustre-OST0000: new disk, initializing [ 213.773280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.780120] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.840418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.818482] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.781733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.790473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.827942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.268252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.391367] Lustre: lustre-OST0001: new disk, initializing [ 233.394449] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.398965] Lustre: 9534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.444189] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.621669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.627708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.659933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 239.013872] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.658719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.522562] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 262.442973] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing check_logdir /tmp/testlogs/ [ 266.740137] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing yml_node [ 271.101300] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 274.135326] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 277.254647] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 278.506735] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 09:33:33 EDT 2026 [ 299.035840] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 301.610430] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.557439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.559792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.575851] Lustre: Skipped 3 previous similar messages [ 323.598973] Lustre: Skipped 3 previous similar messages [ 327.774469] Lustre: server umount lustre-MDT0000 complete [ 328.673088] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.703458] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 331.466616] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800866 with bad export cookie 13779809541337734704 [ 331.476696] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.499969] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.792659] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.793449] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.820079] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 333.846471] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.853301] Lustre: Skipped 1 previous similar message [ 336.801573] LustreError: 8467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.801709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.834207] Lustre: Skipped 1 previous similar message [ 338.351210] Lustre: server umount lustre-MDT0001 complete [ 342.873582] Lustre: server umount lustre-OST0000 complete [ 348.098571] Lustre: server umount lustre-OST0001 complete [ 355.496410] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 364.999493] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 402.225393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.419996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.437737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.507537] Lustre: lustre-MDT0000: new disk, initializing [ 402.565942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.579834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.710883] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 417.669416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.843715] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.907832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.914949] Lustre: Skipped 1 previous similar message [ 418.036449] Lustre: lustre-MDT0001: new disk, initializing [ 418.138795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.170410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.175976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.961601] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 431.110546] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.751168] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 443.315957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.570456] Lustre: lustre-OST0000: new disk, initializing [ 443.574626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.580915] Lustre: 17058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.673088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.513808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.519880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.640882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.830858] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 458.997931] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 463.147891] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 463.388893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.575431] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 468.869576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 471.525097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.531780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.532917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.537891] Lustre: Skipped 2 previous similar messages [ 471.579921] Lustre: Skipped 1 previous similar message [ 476.648656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.655806] Lustre: Skipped 1 previous similar message [ 485.343131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.768077] Lustre: server umount lustre-OST0000 complete [ 492.000922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.017470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.033584] Lustre: Skipped 3 previous similar messages [ 496.376324] Lustre: server umount lustre-MDT0000 complete [ 497.121769] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.133747] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 500.052613] LustreError: 15883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801035 with bad export cookie 13779809541337736237 [ 500.063976] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.067414] LustreError: 15883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.451572] Lustre: server umount lustre-MDT0001 complete [ 504.827727] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:37:18 (1786801038) [ 511.449410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.891957] LustreError: 19323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.984364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.794758] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 522.209982] LustreError: 19324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.222643] LustreError: 19324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 524.905488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.495895] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 536.527266] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 541.765786] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 548.273748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.231801] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 562.021340] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 571.365243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 571.383834] LustreError: Skipped 1 previous similar message [ 571.393778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.412046] Lustre: Skipped 1 previous similar message [ 571.421376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.414880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 575.829919] Lustre: server umount lustre-OST0000 complete [ 581.600329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 581.603911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.630180] Lustre: Skipped 1 previous similar message [ 585.944229] Lustre: server umount lustre-MDT0000 complete [ 588.768991] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.788923] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 589.631524] LustreError: 20063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801124 with bad export cookie 13779809541337737119 [ 589.641248] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.649718] LustreError: 20063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 590.215863] Lustre: server umount lustre-MDT0001 complete [ 601.034290] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 604.256882] Key type lgssc unregistered [ 604.549413] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.556929] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.572602] LNet: Removed LNI 192.168.203.141@tcp [ 605.560137] Key type .llcrypt unregistered [ 605.566546] Key type ._llcrypt unregistered [ 633.085853] Key type ._llcrypt registered [ 633.088116] Key type .llcrypt registered [ 633.175518] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 634.200787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 634.218683] alg: No test for adler32 (adler32-zlib) [ 635.205657] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 635.369033] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 637.015175] Key type lgssc registered [ 638.187752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 645.688447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 645.710942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.371054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.839350] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 656.868765] LustreError: 24278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.897631] LustreError: 24278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 661.989386] LustreError: 24279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.693879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.365684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 667.496294] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 675.646490] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 682.358719] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 689.051343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 689.509932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.594098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 694.999103] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 701.146653] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 718.306844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 718.313743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.328872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.653982] Lustre: server umount lustre-OST0000 complete [ 724.960510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.968739] LustreError: Skipped 1 previous similar message [ 724.971485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.980569] Lustre: Skipped 1 previous similar message [ 724.988417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 724.992564] Lustre: Skipped 1 previous similar message [ 728.546778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.548135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.420279] Lustre: server umount lustre-MDT0000 complete [ 733.664299] LustreError: 26023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.683860] LustreError: 26023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 733.941081] LustreError: 24259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801269 with bad export cookie 14303462634373301070 [ 733.949386] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.952056] LustreError: 24259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 734.405842] Lustre: server umount lustre-MDT0001 complete [ 744.147122] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 746.533860] Key type lgssc unregistered [ 746.808677] LNet: 28257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.815615] LNetError: 28257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.882426] LNet: Removed LNI 192.168.203.141@tcp [ 748.798419] Key type .llcrypt unregistered [ 748.800811] Key type ._llcrypt unregistered [ 773.244476] Key type ._llcrypt registered [ 773.247677] Key type .llcrypt registered [ 773.394390] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 774.556736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.582518] alg: No test for adler32 (adler32-zlib) [ 775.778933] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 776.027536] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 777.727191] Key type lgssc registered [ 778.879726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 786.578846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 786.610390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.082920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.319235] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 798.697683] LustreError: 29671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 798.721196] LustreError: 29671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 799.326591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.625236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 803.203853] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 809.706266] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 814.571640] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 825.742238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 826.019399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.429462] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 833.200779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 838.874597] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 852.967259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 852.980617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.002772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.011817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.017461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.019907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 859.169786] Lustre: server umount lustre-OST0000 complete [ 865.249210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.257624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.259316] Lustre: Skipped 1 previous similar message [ 865.279165] Lustre: Skipped 2 previous similar messages [ 869.165255] Lustre: server umount lustre-MDT0000 complete [ 870.385357] LustreError: 30410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.460855] LustreError: 30410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 873.872763] LustreError: 30404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801409 with bad export cookie 15696408453670935876 [ 873.876535] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.883633] LustreError: 30404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 874.403186] Lustre: server umount lustre-MDT0001 complete [ 886.221172] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 890.018913] Key type lgssc unregistered [ 890.345666] LNet: 33568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.358136] LNetError: 33568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.383245] LNet: Removed LNI 192.168.203.141@tcp [ 891.186362] Key type .llcrypt unregistered [ 891.195513] Key type ._llcrypt unregistered [ 916.042361] Key type ._llcrypt registered [ 916.048377] Key type .llcrypt registered [ 916.172370] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 916.953674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 916.971958] alg: No test for adler32 (adler32-zlib) [ 918.095138] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 918.354123] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 920.015189] Key type lgssc registered [ 921.145900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 928.521967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 928.533232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.022279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.266191] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 940.516720] LustreError: 34982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 940.535445] LustreError: 34982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 942.974749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.373553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.373063] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 956.967388] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 964.488835] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 970.729633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 971.098153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.213274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 976.659566] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 983.832476] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 997.861375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 997.867528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.895328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.911261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.911466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.939878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.655789] Lustre: server umount lustre-OST0000 complete [ 1010.144527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.146136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1010.151251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.151260] Lustre: Skipped 1 previous similar message [ 1013.278941] Lustre: server umount lustre-MDT0000 complete [ 1015.267241] LustreError: 34983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.279025] LustreError: 34983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1017.553196] LustreError: 36825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801552 with bad export cookie 6581844497511523876 [ 1017.557835] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.573157] LustreError: 36825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1018.055461] Lustre: server umount lustre-MDT0001 complete [ 1032.551557] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1036.843525] Key type lgssc unregistered [ 1037.078511] LNet: 38539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.103975] LNetError: 38539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.124759] LNet: Removed LNI 192.168.203.141@tcp [ 1037.899683] Key type .llcrypt unregistered [ 1037.906942] Key type ._llcrypt unregistered [ 1064.742262] Key type ._llcrypt registered [ 1064.744251] Key type .llcrypt registered [ 1064.848428] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1065.566236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1065.596150] alg: No test for adler32 (adler32-zlib) [ 1066.664648] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1066.839580] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1068.471170] Key type lgssc registered [ 1069.782296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.958403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1077.989850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.338188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.949485] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1089.509985] LustreError: 39953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.531549] LustreError: 39953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1092.476206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.926398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.333776] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1104.157567] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1109.826111] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1118.358435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1119.054914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.171662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1126.630425] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1133.276912] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.758885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.768101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.784643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.793218] Lustre: Skipped 1 previous similar message [ 1144.453680] Lustre: server umount lustre-OST0000 complete [ 1163.235964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.257486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.265137] Lustre: Skipped 1 previous similar message [ 1164.321055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.334310] Lustre: Skipped 2 previous similar messages [ 1168.934787] Lustre: server umount lustre-MDT0000 complete [ 1169.385757] LustreError: 39953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.398878] LustreError: 39953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1171.630717] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801706 with bad export cookie 13341391045576813525 [ 1171.636104] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.640978] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1177.952869] Lustre: server umount lustre-MDT0001 complete [ 1187.336710] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1190.004180] Key type lgssc unregistered [ 1190.307854] LNet: 43661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.315079] LNetError: 43661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.326866] LNet: Removed LNI 192.168.203.141@tcp [ 1191.008744] Key type .llcrypt unregistered [ 1191.017844] Key type ._llcrypt unregistered [ 1212.973598] Key type ._llcrypt registered [ 1212.975564] Key type .llcrypt registered [ 1213.069796] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1214.055616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.072538] alg: No test for adler32 (adler32-zlib) [ 1215.077787] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1215.290523] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1216.959534] Key type lgssc registered [ 1217.895680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1224.384421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1224.417514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.820509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.847703] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1234.920715] LustreError: 45074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.938581] LustreError: 45074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1236.917964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.202360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1240.712676] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1248.464592] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1254.065452] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1262.105499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1262.370168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.249660] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1270.578508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1276.017064] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1288.137347] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 1288.161949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1288.173273] Lustre: Skipped 2 previous similar messages [ 1291.583687] Lustre: server umount lustre-MDT0000 complete [ 1293.240608] LustreError: 45070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.125616] LustreError: 45054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801831 with bad export cookie 11855525965932606201 [ 1296.139195] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.144647] LustreError: 45054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1298.354499] LustreError: 47369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.390138] LustreError: 47369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1298.407085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.422273] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (stopping) [ 1298.445133] Lustre: Skipped 4 previous similar messages [ 1301.409361] LustreError: 45069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.409434] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1302.538351] Lustre: server umount lustre-MDT0001 complete [ 1318.873269] Lustre: server umount lustre-OST0000 complete [ 1336.832986] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1339.715550] Key type lgssc unregistered [ 1340.065648] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.076143] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.089252] LNet: Removed LNI 192.168.203.141@tcp [ 1340.660102] Key type .llcrypt unregistered [ 1340.661539] Key type ._llcrypt unregistered [ 1362.851966] Key type ._llcrypt registered [ 1362.854720] Key type .llcrypt registered [ 1362.974944] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1364.603656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.637442] alg: No test for adler32 (adler32-zlib) [ 1365.904409] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1366.176539] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1367.898344] Key type lgssc registered [ 1369.191016] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.713605] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1377.724712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.248426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1398.432804] LustreError: 50299:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 1425.121018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.319257] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1439.629377] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1607.796169] Lustre: server umount lustre-OST0000 complete [ 1622.653752] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1625.357429] Key type lgssc unregistered [ 1625.651899] LNet: 51975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.678817] LNetError: 51975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.694374] LNet: Removed LNI 192.168.203.141@tcp [ 1626.537708] Key type .llcrypt unregistered [ 1626.542625] Key type ._llcrypt unregistered [ 1649.633967] Key type ._llcrypt registered [ 1649.637816] Key type .llcrypt registered [ 1649.789117] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1650.874977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.952247] alg: No test for adler32 (adler32-zlib) [ 1652.023260] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1652.225844] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1653.871572] Key type lgssc registered [ 1654.747945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.941629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.961273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.352617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.991985] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1673.734860] LustreError: 53390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.748351] LustreError: 53390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1677.314560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.913346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1682.709894] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1690.172307] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.478389] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1704.367783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.941930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.016274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1711.887159] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1719.093579] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1722.344451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.347914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.384107] Lustre: Skipped 1 previous similar message [ 1727.471814] LustreError: 55214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.500016] LustreError: 55214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1727.730229] Lustre: server umount lustre-OST0000 complete [ 1732.580239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.581755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.586249] Lustre: Skipped 1 previous similar message [ 1737.637624] Lustre: server umount lustre-MDT0000 complete [ 1737.696720] LustreError: 53390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.708211] LustreError: 53390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1741.112583] LustreError: 53371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802276 with bad export cookie 15789206092135392647 [ 1741.121923] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.128592] LustreError: 53371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1741.524939] Lustre: server umount lustre-MDT0001 complete [ 1758.955723] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1763.760726] Key type lgssc unregistered [ 1764.595563] LNet: 56922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.639831] LNetError: 56922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.672594] LNet: Removed LNI 192.168.203.141@tcp [ 1766.380395] Key type .llcrypt unregistered [ 1766.382472] Key type ._llcrypt unregistered [ 1790.642437] Key type ._llcrypt registered [ 1790.644635] Key type .llcrypt registered [ 1790.776079] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 1792.048643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.107315] alg: No test for adler32 (adler32-zlib) [ 1793.241150] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1793.525358] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 1795.300281] Key type lgssc registered [ 1796.652939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.821478] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.836173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.384112] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1825.503967] LustreError: 58310:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 1852.191210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.546854] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1867.612127] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1874.677129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.273326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.354529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1879.515806] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1880.544909] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.581592] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1885.669162] LustreError: 58336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.717596] LustreError: 59344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.744858] LustreError: 59344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1888.151291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.655751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1893.299724] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 1900.415711] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1908.237741] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1911.265840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.268532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.283140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.337092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.349592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.358299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.379749] Lustre: server umount lustre-OST0000 complete [ 1943.011622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.017922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.036536] Lustre: Skipped 1 previous similar message [ 1943.055729] Lustre: Skipped 2 previous similar messages [ 1948.133927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.143662] Lustre: Skipped 1 previous similar message [ 1953.252836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.343131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1957.550520] Lustre: server umount lustre-MDT0000 complete [ 1959.396301] LustreError: 59352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.409052] LustreError: 59352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1961.506978] LustreError: 58316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802496 with bad export cookie 5693675699170774806 [ 1961.526218] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.067609] Lustre: server umount lustre-MDT0001 complete [ 1981.181902] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 1984.201269] Key type lgssc unregistered [ 1984.586796] LNet: 62072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.592760] LNetError: 62072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.607651] LNet: Removed LNI 192.168.203.141@tcp [ 1985.292667] Key type .llcrypt unregistered [ 1985.294892] Key type ._llcrypt unregistered [ 2008.316961] Key type ._llcrypt registered [ 2008.319155] Key type .llcrypt registered [ 2008.400241] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2009.669253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.694414] alg: No test for adler32 (adler32-zlib) [ 2010.736251] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2011.130655] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2012.832878] Key type lgssc registered [ 2014.139801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2021.593654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2021.617139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.181973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.739740] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2033.636438] LustreError: 63485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.665975] LustreError: 63485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2037.931614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.353875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.436748] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2049.944285] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2055.630990] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2062.569191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2062.950930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.089925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2068.727357] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2076.283557] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2094.049600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.050251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.071374] Lustre: Skipped 1 previous similar message [ 2094.092450] Lustre: Skipped 1 previous similar message [ 2097.817173] Lustre: server umount lustre-OST0000 complete [ 2104.292097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.319603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.337176] Lustre: Skipped 1 previous similar message [ 2108.013240] Lustre: server umount lustre-MDT0000 complete [ 2109.414742] LustreError: 63484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.432763] LustreError: 63484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2111.603465] LustreError: 63466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802646 with bad export cookie 11695607194338032237 [ 2111.611812] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.619712] LustreError: 63466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2112.144125] Lustre: server umount lustre-MDT0001 complete [ 2126.291204] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2129.958891] Key type lgssc unregistered [ 2130.381651] LNet: 67040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.393540] LNetError: 67040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.438886] LNet: Removed LNI 192.168.203.141@tcp [ 2131.456887] Key type .llcrypt unregistered [ 2131.462405] Key type ._llcrypt unregistered [ 2154.250376] Key type ._llcrypt registered [ 2154.254375] Key type .llcrypt registered [ 2154.420679] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2155.393268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.496738] alg: No test for adler32 (adler32-zlib) [ 2156.548762] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2156.858452] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2158.568459] Key type lgssc registered [ 2159.961386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.194548] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:05:03 (1786802703) [ 2176.008685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.027420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.435597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.313253] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2187.744533] LustreError: 68724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.771641] LustreError: 68724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2191.526179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.069575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.470698] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2204.840796] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.922913] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2217.323694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2217.668931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.829276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2223.465403] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2230.262264] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2239.043233] Lustre: Failing over lustre-MDT0000 [ 2239.255094] Lustre: server umount lustre-MDT0000 complete [ 2242.514376] LustreError: 68719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.542790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.586071] Lustre: Skipped 2 previous similar messages [ 2247.612544] LustreError: 69459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.652080] LustreError: 69459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2248.461397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.587517] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.722671] Lustre: *** cfs_fail_loc=135, val=0*** [ 2248.866518] LustreError: 71811:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2248.873857] LustreError: 71811:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.141@tcp: cfg command failed: rc = -2 [ 2248.898719] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2248.898719] [ 2248.918895] LustreError: MGC192.168.203.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2248.937276] LustreError: 71800:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2248.945955] LustreError: 71800:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2248.957497] LustreError: 71800:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2248.966732] LustreError: 71800:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2249.039229] Lustre: server umount lustre-MDT0000 complete [ 2249.048454] LustreError: 71800:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2252.736735] LustreError: 68720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.864723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.024462] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.269583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.307484] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2257.855470] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2260.696664] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2261.484442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2261.556295] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2261.596708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2268.128054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.132279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.139541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.725398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.737551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.532632] Lustre: server umount lustre-OST0000 complete [ 2281.957397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.959550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2281.967288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.988894] Lustre: Skipped 2 previous similar messages [ 2283.242951] Lustre: server umount lustre-MDT0000 complete [ 2286.571048] LustreError: 68706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802821 with bad export cookie 9977770626395136305 [ 2286.584643] LustreError: 68706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2286.587088] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.869206] Lustre: server umount lustre-MDT0001 complete [ 2298.609818] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2301.514439] Key type lgssc unregistered [ 2301.789820] LNet: 73589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.798182] LNetError: 73589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.818279] LNet: Removed LNI 192.168.203.141@tcp [ 2302.941308] Key type .llcrypt unregistered [ 2302.943431] Key type ._llcrypt unregistered [ 2327.396847] Key type ._llcrypt registered [ 2327.406858] Key type .llcrypt registered [ 2327.515975] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2328.620772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.632874] alg: No test for adler32 (adler32-zlib) [ 2329.608438] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2329.986727] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2331.615731] Key type lgssc registered [ 2332.828190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2341.757805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2341.780297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.598299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.468530] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2353.126729] LustreError: 74996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.134334] LustreError: 74996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2357.621850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.859103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.135683] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2372.762823] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2378.480173] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2386.741973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.288960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.361319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2395.128993] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2405.567953] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2411.386909] Lustre: Failing over lustre-MDT0000 [ 2411.660964] Lustre: server umount lustre-MDT0000 complete [ 2412.998735] LustreError: 74993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.030581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.045121] Lustre: Skipped 2 previous similar messages [ 2418.097401] LustreError: 74992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.120319] LustreError: 74992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2419.492280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.590910] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.684993] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2419.805699] LustreError: 78080:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2419.813890] LustreError: 78080:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.141@tcp: cfg command failed: rc = -9 [ 2419.829906] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2419.829906] [ 2419.845851] LustreError: MGC192.168.203.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2419.868075] LustreError: 78069:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2419.888925] LustreError: 78069:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2419.905290] LustreError: 78069:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2419.911232] LustreError: 78069:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2419.963482] Lustre: server umount lustre-MDT0000 complete [ 2419.968676] LustreError: 78069:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2423.223668] LustreError: 74991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.239830] LustreError: 74991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2426.645120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.863421] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.872463] LustreError: 74988:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2426.878023] LustreError: 74306:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ef3a3648 x1873598601002624/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2427.082796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.122837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2428.340940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2430.859862] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2432.506303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2432.550765] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2432.619644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2438.625399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2438.632609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.669889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.727651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.744109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.747687] Lustre: server umount lustre-OST0000 complete [ 2452.961994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.968459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2452.977889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.977899] Lustre: Skipped 1 previous similar message [ 2452.983538] Lustre: Skipped 1 previous similar message [ 2454.949062] LustreError: 79109:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000a472b87f) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2455.026149] Lustre: server umount lustre-MDT0000 complete [ 2458.090694] LustreError: 75948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.108229] LustreError: 75948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2459.290934] LustreError: 74977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802994 with bad export cookie 13516166909537030437 [ 2459.293655] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.303152] LustreError: 74977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2459.797197] Lustre: server umount lustre-MDT0001 complete [ 2472.800323] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2476.497339] Key type lgssc unregistered [ 2476.804680] LNet: 79861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2476.829810] LNetError: 79861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.855910] LNet: Removed LNI 192.168.203.141@tcp [ 2478.026913] Key type .llcrypt unregistered [ 2478.029250] Key type ._llcrypt unregistered [ 2491.224970] Key type ._llcrypt registered [ 2491.229724] Key type .llcrypt registered [ 2491.335321] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2491.677382] Key type .llcrypt unregistered [ 2491.684593] Key type ._llcrypt unregistered [ 2514.616626] Key type ._llcrypt registered [ 2514.621573] Key type .llcrypt registered [ 2514.774841] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2516.190723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.201731] alg: No test for adler32 (adler32-zlib) [ 2517.353777] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2517.666663] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2519.336147] Key type lgssc registered [ 2520.472755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.302189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2528.311233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.599341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.187743] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2538.977522] LustreError: 81766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.007324] LustreError: 81766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2542.508682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.889911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2546.481315] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2553.792523] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2558.628153] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2565.016707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.454409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.426447] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2574.707770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2578.350058] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2588.128700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.142039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.163859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.185272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.186123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.208031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.992749] Lustre: server umount lustre-OST0000 complete [ 2599.392989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2599.400312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.415320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2599.420321] Lustre: Skipped 1 previous similar message [ 2601.800726] Lustre: server umount lustre-MDT0000 complete [ 2604.923524] LustreError: 83606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803140 with bad export cookie 8155949678047614351 [ 2604.925113] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.929186] LustreError: 83606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2605.226448] Lustre: server umount lustre-MDT0001 complete [ 2615.042563] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2617.814593] Key type lgssc unregistered [ 2618.074553] LNet: 85295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2618.079332] LNetError: 85295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2618.092737] LNet: Removed LNI 192.168.203.141@tcp [ 2618.880403] Key type .llcrypt unregistered [ 2618.882374] Key type ._llcrypt unregistered [ 2641.324698] Key type ._llcrypt registered [ 2641.326651] Key type .llcrypt registered [ 2641.413873] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2642.395117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2642.437810] alg: No test for adler32 (adler32-zlib) [ 2643.448901] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2643.632356] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2645.271169] Key type lgssc registered [ 2646.780579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2656.104586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2656.130347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.705909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.046872] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2666.979520] LustreError: 86708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.997431] LustreError: 86708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2671.022245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.216595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.852916] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2683.174480] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2689.814928] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2696.518125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.913536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.101096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2703.436901] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2709.600537] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.719320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2718.729508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.749135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2721.898918] Lustre: server umount lustre-OST0000 complete [ 2727.396246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2727.400232] LustreError: Skipped 1 previous similar message [ 2727.404170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.414653] Lustre: Skipped 1 previous similar message [ 2727.422462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2727.438158] Lustre: Skipped 1 previous similar message [ 2728.928836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.936778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.955845] Lustre: Skipped 1 previous similar message [ 2732.511607] Lustre: server umount lustre-MDT0000 complete [ 2734.048794] LustreError: 86709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.062760] LustreError: 86709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2736.010212] LustreError: 86691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803271 with bad export cookie 10697164951919698435 [ 2736.019113] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2736.027920] LustreError: 86691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2736.322357] Lustre: server umount lustre-MDT0001 complete [ 2747.152871] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2750.016587] Key type lgssc unregistered [ 2750.228941] LNet: 90241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2750.235542] LNetError: 90241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.274958] LNet: Removed LNI 192.168.203.141@tcp [ 2752.087175] Key type .llcrypt unregistered [ 2752.089669] Key type ._llcrypt unregistered [ 2781.159573] Key type ._llcrypt registered [ 2781.162560] Key type .llcrypt registered [ 2781.263868] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2782.155106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.181821] alg: No test for adler32 (adler32-zlib) [ 2783.193451] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2783.401885] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2785.119165] Key type lgssc registered [ 2786.294255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.801621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2792.839520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.215260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.835314] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2804.707956] LustreError: 91661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.733666] LustreError: 91661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2805.744774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.990357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.081413] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2817.283325] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2823.838409] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2830.860671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.592214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.766639] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2839.806405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2845.882771] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2858.466503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.474247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.486969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.978971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2859.005550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.024072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.522314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.541696] Lustre: Skipped 1 previous similar message [ 2862.432783] Lustre: server umount lustre-OST0000 complete [ 2867.680498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2867.688806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.706300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.760532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.093974] Lustre: server umount lustre-MDT0000 complete [ 2875.043675] LustreError: 94441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803410 with bad export cookie 6605939214050079367 [ 2875.052368] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.056595] LustreError: 94441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2875.300365] Lustre: server umount lustre-MDT0001 complete [ 2883.680882] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 2886.017452] Key type lgssc unregistered [ 2886.261175] LNet: 95191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2886.269646] LNetError: 95191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2886.282111] LNet: Removed LNI 192.168.203.141@tcp [ 2886.958503] Key type .llcrypt unregistered [ 2886.960521] Key type ._llcrypt unregistered [ 2905.057835] Key type ._llcrypt registered [ 2905.061231] Key type .llcrypt registered [ 2905.149399] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 2905.827776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2905.972792] alg: No test for adler32 (adler32-zlib) [ 2906.973994] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 2907.169405] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 2908.807284] Key type lgssc registered [ 2909.631834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.580589] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.601319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.103515] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2936.227739] LustreError: 96556:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 2962.847197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.689194] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 2976.630514] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2982.451535] Lustre: server umount lustre-OST0000 complete [ 2993.054677] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:18:47 (1786803527) [ 3001.298477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.725674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.672686] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3006.951966] LustreError: 98198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.977932] LustreError: 98198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3011.049914] LustreError: 98199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.649410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.029736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.975395] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3029.285238] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3035.308948] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3042.142707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3042.585736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.716846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3049.073821] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3056.877707] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3071.463584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.464979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3071.468666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3071.480054] Lustre: Skipped 1 previous similar message [ 3075.425279] Lustre: server umount lustre-OST0000 complete [ 3081.696635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.700355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.714857] Lustre: Skipped 1 previous similar message [ 3081.720960] Lustre: Skipped 2 previous similar messages [ 3084.589060] Lustre: server umount lustre-MDT0000 complete [ 3086.823440] LustreError: 98194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.835822] LustreError: 98194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3087.425322] LustreError: 98932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803622 with bad export cookie 15489166630650345894 [ 3087.441546] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.443315] LustreError: 98932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3087.796916] Lustre: server umount lustre-MDT0001 complete [ 3096.764534] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 3098.950272] Key type lgssc unregistered [ 3099.251960] LNet: 101893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3099.257305] LNetError: 101893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3099.271540] LNet: Removed LNI 192.168.203.141@tcp [ 3099.941351] Key type .llcrypt unregistered [ 3099.945410] Key type ._llcrypt unregistered [ 3133.754214] Key type ._llcrypt registered [ 3133.756795] Key type .llcrypt registered [ 3133.830828] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3134.621031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3134.647870] alg: No test for adler32 (adler32-zlib) [ 3135.743642] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 3135.911822] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 3137.535197] Key type lgssc registered [ 3138.583546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3145.683911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3145.707915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.044041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.154955] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3157.491929] LustreError: 104056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3157.526803] LustreError: 104056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3160.138283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.487615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.880476] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3172.570557] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3179.009086] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3186.540545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3186.872538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.006851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3193.845764] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3202.008877] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3213.792168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3213.800419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.813169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.840125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.843782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.848980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.865364] Lustre: Skipped 1 previous similar message [ 3218.902454] Lustre: server umount lustre-OST0000 complete [ 3226.082232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.096293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.107903] Lustre: Skipped 1 previous similar message [ 3228.838924] Lustre: server umount lustre-MDT0000 complete [ 3231.209594] LustreError: 104784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.246109] LustreError: 104784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3232.429721] LustreError: 104038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803767 with bad export cookie 14515426781707717257 [ 3232.432888] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.441858] LustreError: 104038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3232.714729] Lustre: server umount lustre-MDT0001 complete [ 3241.498149] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 3243.927817] Key type lgssc unregistered [ 3244.169709] LNet: 107588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.175166] LNetError: 107588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.199828] LNet: Removed LNI 192.168.203.141@tcp [ 3244.833224] Key type .llcrypt unregistered [ 3244.837070] Key type ._llcrypt unregistered [ 3264.964214] Key type ._llcrypt registered [ 3264.968166] Key type .llcrypt registered [ 3265.148509] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3266.177745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3266.200355] alg: No test for adler32 (adler32-zlib) [ 3267.207851] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 3267.476407] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 3269.263161] Key type lgssc registered [ 3270.354951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3276.793836] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3276.807113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3292.319435] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3297.440645] LustreError: 108831:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 3324.191245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.525909] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3338.060996] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3344.581881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.829075] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3344.854946] Lustre: MGS: Client 9d89eb41-7d3c-4580-adee-af89bc79d742 (at 0@lo) reconnecting [ 3344.866773] Lustre: MGC192.168.203.141@tcp: Connection restored to 0@lo (at 0@lo) [ 3345.048940] LustreError: MGC192.168.203.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3345.060847] LustreError: 109857:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3345.071418] LustreError: 109857:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3345.079489] LustreError: 109857:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3345.084036] LustreError: 108837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803880 with bad export cookie 14937180678077616220 [ 3345.098357] LustreError: 108837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3345.169382] Lustre: server umount lustre-MDT0000 complete [ 3345.177487] LustreError: 109857:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3373.210690] Lustre: server umount lustre-OST0000 complete [ 3382.417616] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 3392.106673] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3427.439386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.729501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3427.772736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3427.911503] Lustre: lustre-MDT0000: new disk, initializing [ 3428.023522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.047137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3434.411281] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3444.162918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.312459] Lustre: 112661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3444.385029] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3444.390343] Lustre: Skipped 1 previous similar message [ 3444.491829] Lustre: lustre-MDT0001: new disk, initializing [ 3444.604205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3444.658559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3444.669104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3449.782518] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3458.762265] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.313730] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.108242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.309546] Lustre: lustre-OST0000: new disk, initializing [ 3468.312928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3468.318713] Lustre: 114625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3468.371568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.900300] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3474.996156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3475.009666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3475.086556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3480.714365] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3484.641820] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3484.827433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3488.529516] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3488.710911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3490.784724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3490.787748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.796585] LustreError: Skipped 1 previous similar message [ 3490.804060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.909897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.918632] Lustre: Skipped 2 previous similar messages [ 3496.203484] Lustre: server umount lustre-OST0000 complete [ 3501.026545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.029711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.037675] Lustre: Skipped 1 previous similar message [ 3501.040726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.066428] Lustre: Skipped 1 previous similar message [ 3506.144613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3506.156386] Lustre: Skipped 1 previous similar message [ 3511.265212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3511.274076] Lustre: Skipped 1 previous similar message [ 3514.335445] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3514.523385] Lustre: server umount lustre-MDT0000 complete [ 3517.589505] LustreError: 114624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804052 with bad export cookie 14937180678077616332 [ 3517.597645] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.607254] LustreError: 114624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3517.627695] LustreError: Skipped 1 previous similar message [ 3517.875684] Lustre: server umount lustre-MDT0001 complete [ 3525.588986] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:27:40 (1786804060) [ 3528.336414] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3541.457994] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 3551.058537] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3587.020858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.255670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3587.280900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3587.346584] Lustre: lustre-MDT0000: new disk, initializing [ 3587.403823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.415897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3591.327988] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3601.714993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.886809] Lustre: 119523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3601.954885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3601.963731] Lustre: Skipped 1 previous similar message [ 3602.047269] Lustre: lustre-MDT0001: new disk, initializing [ 3602.157414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.194609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3602.213659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3606.911864] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3615.737333] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3622.670620] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3630.463475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.971628] Lustre: lustre-OST0000: new disk, initializing [ 3630.992724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3631.007535] Lustre: 121491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3631.174416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.621547] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3640.924633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3640.946873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3641.160983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3646.628786] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3652.694871] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3652.997577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3658.003604] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3658.243779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3661.797360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3661.799141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.806765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3661.811337] LustreError: Skipped 1 previous similar message [ 3661.855609] Lustre: Skipped 2 previous similar messages [ 3666.762589] Lustre: server umount lustre-OST0000 complete [ 3672.042354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.584083] Lustre: server umount lustre-MDT0000 complete [ 3681.292570] LustreError: 120326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804216 with bad export cookie 14937180678077617214 [ 3681.296726] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3681.305682] LustreError: 120326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3681.730166] Lustre: server umount lustre-MDT0001 complete [ 3690.787703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.295105] LustreError: 123710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.326773] LustreError: 123710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3691.425323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.846518] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3696.622401] LustreError: 123711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.732182] LustreError: 123710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.772028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.013756] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3715.993523] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.301395] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3727.650992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3734.044957] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3742.158157] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3748.925466] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3756.000579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3756.011035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.024980] Lustre: Skipped 1 previous similar message [ 3756.029407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.037489] Lustre: Skipped 5 previous similar messages [ 3758.297505] Lustre: server umount lustre-OST0000 complete [ 3761.638245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3761.649537] LustreError: Skipped 1 previous similar message [ 3768.029084] Lustre: server umount lustre-MDT0000 complete [ 3770.896434] LustreError: 123692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804306 with bad export cookie 14937180678077618096 [ 3770.905726] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.911164] LustreError: 123692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3771.153653] Lustre: server umount lustre-MDT0001 complete [ 3783.352403] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 3786.506905] Key type lgssc unregistered [ 3786.787634] LNet: 127339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.800431] LNetError: 127339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3786.843027] LNet: Removed LNI 192.168.203.141@tcp [ 3787.584933] Key type .llcrypt unregistered [ 3787.586360] Key type ._llcrypt unregistered [ 3806.079564] Key type ._llcrypt registered [ 3806.082916] Key type .llcrypt registered [ 3806.245472] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 3822.070736] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 3822.864194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.898270] alg: No test for adler32 (adler32-zlib) [ 3823.938557] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 3824.185453] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 3825.847609] Key type lgssc registered [ 3826.731866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3860.747186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.778288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.936024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.955752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3862.056389] Lustre: lustre-MDT0000: new disk, initializing [ 3862.148118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.175063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3866.117342] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3877.176442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.353962] Lustre: 130493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3877.412668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3877.419673] Lustre: Skipped 1 previous similar message [ 3877.555278] Lustre: lustre-MDT0001: new disk, initializing [ 3877.689712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.743647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3877.769353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3882.324995] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3890.490067] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3895.457484] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3901.495582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.709506] Lustre: lustre-OST0000: new disk, initializing [ 3901.713867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3901.719306] Lustre: 132461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3901.833936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.022692] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3908.665497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3908.685508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3908.806739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3916.094383] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3919.931160] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3920.136246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3924.585930] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3924.816915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3929.581026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3929.588142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.598607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.618449] Lustre: Skipped 1 previous similar message [ 3929.628747] Lustre: Skipped 1 previous similar message [ 3932.991498] Lustre: server umount lustre-OST0000 complete [ 3939.297096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3939.309124] LustreError: Skipped 1 previous similar message [ 3939.313970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.324392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.638839] Lustre: server umount lustre-MDT0000 complete [ 3946.090803] LustreError: 132460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804481 with bad export cookie 13238823642380183331 [ 3946.099284] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.106021] LustreError: 132460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3946.528989] Lustre: server umount lustre-MDT0001 complete [ 3954.602556] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:34:49 (1786804489) [ 3960.899224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3961.215180] LustreError: 134872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.235123] LustreError: 134872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3961.269397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.579397] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3966.441410] LustreError: 134873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.557079] LustreError: 134872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.380308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.794445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3977.508608] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 3984.280427] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3989.670964] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3991.522062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.525605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3991.542177] Lustre: Skipped 2 previous similar messages [ 3991.553710] Lustre: Skipped 1 previous similar message [ 3996.642678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3996.650819] Lustre: Skipped 2 previous similar messages [ 3997.755934] Lustre: server umount lustre-MDT0000 complete [ 3998.689033] LustreError: 134869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.714532] LustreError: 134869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4001.339458] LustreError: 134854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804536 with bad export cookie 13238823642380184206 [ 4001.352793] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4001.652694] Lustre: server umount lustre-MDT0001 complete [ 4009.027379] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:35:43 (1786804543) [ 4014.893324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.431842] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4035.551675] LustreError: 137287:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 4061.279437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.495784] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4074.506273] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4076.721467] Lustre: server umount lustre-OST0000 complete [ 4084.557355] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:36:59 (1786804619) [ 4090.637431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.007834] LustreError: 138734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.099429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.922493] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4101.601376] LustreError: 138734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.624622] LustreError: 138734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4102.364321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.775278] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4113.441986] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4118.810937] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4124.585204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4124.853967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.859558] Lustre: Skipped 1 previous similar message [ 4130.652433] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4138.131538] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4146.661143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.673254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.680170] Lustre: Skipped 2 previous similar messages [ 4146.687485] Lustre: Skipped 1 previous similar message [ 4152.128245] Lustre: server umount lustre-MDT0000 complete [ 4155.427217] LustreError: 138716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804690 with bad export cookie 13238823642380184878 [ 4155.438172] LustreError: 138716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4155.452688] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.699444] Lustre: server umount lustre-MDT0001 complete [ 4169.783046] Lustre: server umount lustre-OST0000 complete [ 4178.382314] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:38:32 (1786804712) [ 4184.199641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.793155] LustreError: 142386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4188.272671] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4197.319404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4197.721403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.732889] Lustre: Skipped 1 previous similar message [ 4201.605178] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4209.586462] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4215.358076] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4221.631881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4228.120482] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4229.119402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4235.804000] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4241.907638] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4242.174652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4247.759923] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4247.932316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4253.667330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4253.674107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.693755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.699420] Lustre: Skipped 4 previous similar messages [ 4255.951325] Lustre: server umount lustre-OST0000 complete [ 4264.416527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4264.420789] LustreError: Skipped 1 previous similar message [ 4264.430859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.454388] Lustre: Skipped 2 previous similar messages [ 4265.966507] Lustre: server umount lustre-MDT0000 complete [ 4270.052405] LustreError: 142386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.082937] LustreError: 142386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4270.370966] LustreError: 142366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804805 with bad export cookie 13238823642380185949 [ 4270.408255] LustreError: 142366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4270.433456] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4270.808551] Lustre: server umount lustre-MDT0001 complete [ 4282.194095] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:40:16 (1786804816) [ 4291.211137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.783586] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4311.906053] LustreError: 146545:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 4337.759950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.769693] Lustre: Skipped 1 previous similar message [ 4344.623430] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4353.117691] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4361.570899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.856770] LustreError: 146570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.949152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4366.692597] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4374.546239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.051291] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4385.780316] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4391.312266] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4396.011690] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4396.199785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4400.077987] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4400.314080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4405.729326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4405.731876] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.736180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.736190] Lustre: Skipped 5 previous similar messages [ 4405.761805] Lustre: Skipped 2 previous similar messages [ 4408.287469] Lustre: server umount lustre-MDT0000 complete [ 4411.409495] LustreError: 146551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804946 with bad export cookie 13238823642380186971 [ 4411.425380] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4411.428392] LustreError: 146551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4411.712900] Lustre: server umount lustre-MDT0001 complete [ 4425.498818] Lustre: server umount lustre-OST0000 complete [ 4432.535194] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:42:47 (1786804967) [ 4438.215949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.791705] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4458.911521] LustreError: 150739:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 4490.855545] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4498.040792] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4504.747355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.148927] LustreError: 150764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.167249] LustreError: 150764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4505.320943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4509.206911] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4517.275796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4521.937392] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4528.866471] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4534.686469] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4541.037035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4541.165066] Lustre: lustre-OST0001: new disk, initializing [ 4541.168858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4541.173868] Lustre: 153612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4548.087234] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4550.170832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4550.177351] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4550.251817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4557.795024] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4561.402972] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4561.555506] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4565.414241] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4565.628911] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4569.056721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4569.072865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.096090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4569.104129] Lustre: Skipped 2 previous similar messages [ 4573.445212] Lustre: server umount lustre-OST0000 complete [ 4583.192545] Lustre: server umount lustre-OST0001 complete [ 4589.537334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4589.542611] LustreError: Skipped 2 previous similar messages [ 4592.535474] Lustre: server umount lustre-MDT0000 complete [ 4595.598654] LustreError: 150745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805130 with bad export cookie 13238823642380187923 [ 4595.607291] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4595.612833] LustreError: 150745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4595.840858] Lustre: server umount lustre-MDT0001 complete [ 4618.888593] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:45:53 (1786805153) [ 4620.842567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4622.228242] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4623.620575] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:45:58 (1786805158) [ 4630.154960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4630.524636] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4630.583504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4630.981605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.987459] Lustre: Skipped 6 previous similar messages [ 4635.726642] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4643.643121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.735816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4643.780343] Lustre: 157171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4648.558493] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4655.409367] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.108238] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4667.337327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.471105] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4673.156709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4673.523960] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4680.270461] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4684.468317] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4684.661679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4688.377465] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4688.600330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4693.477908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4693.494507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.507506] Lustre: Skipped 5 previous similar messages [ 4693.531162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4693.540609] Lustre: Skipped 7 previous similar messages [ 4696.534123] Lustre: server umount lustre-OST0000 complete [ 4716.622373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4722.715929] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4729.652825] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4734.246360] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.254581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.255353] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4734.466412] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4734.622889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4738.896150] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4739.237611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4744.385731] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b3e10c5f800.ost_server_uuid 50 [ 4746.443725] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b3e10c5f800.ost_server_uuid in FULL state after 0 sec [ 4746.753218] LustreError: 157179:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.41@tcp) returned error from blocking AST (req@0000000070148777 x1873600167797504 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005ffdee7a/0xb7b9bff5e01d426d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.41@tcp remote: 0x1b8632cf414a06ba expref: 5 pid: 157926 timeout: 4846 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4746.812974] LustreError: lustre-MDT0000: A client on nid 192.168.203.41@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4746.824980] LustreError: 157168:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.41@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005ffdee7a/0xb7b9bff5e01d426d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.41@tcp remote: 0x1b8632cf414a06ba expref: 6 pid: 157926 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4757.989354] Lustre: server umount lustre-OST0000 complete [ 4768.853132] Lustre: server umount lustre-MDT0000 complete [ 4770.273678] LustreError: 157184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.298111] LustreError: 157184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4772.635394] LustreError: 157163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805307 with bad export cookie 13238823642380189106 [ 4772.640919] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4772.645017] LustreError: 157163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4779.019493] Lustre: server umount lustre-MDT0001 complete [ 4791.038517] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 4793.922712] Key type lgssc unregistered [ 4794.245654] LNet: 163286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4794.252036] LNetError: 163286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4794.265555] LNet: Removed LNI 192.168.203.141@tcp [ 4795.092924] Key type .llcrypt unregistered [ 4795.094742] Key type ._llcrypt unregistered [ 4816.966578] Key type ._llcrypt registered [ 4816.968665] Key type .llcrypt registered [ 4817.045808] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 4818.157550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4818.181099] alg: No test for adler32 (adler32-zlib) [ 4819.242479] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4819.451427] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 4821.107376] Key type lgssc registered [ 4822.438730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4831.106212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4831.144373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.447373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.825718] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4841.956806] LustreError: 164698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4841.976723] LustreError: 164698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4846.401257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.651905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.990648] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4856.288632] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4861.642126] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4867.616357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4867.862935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.246845] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4877.106188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4881.335563] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4887.593172] Lustre: Failing over lustre-MDT0000 [ 4888.075055] Lustre: server umount lustre-MDT0000 complete [ 4888.584555] LustreError: 164723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.645115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.651271] LustreError: 164698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.665228] Lustre: Skipped 2 previous similar messages [ 4892.679087] LustreError: 164698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4897.761474] LustreError: 165428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.770148] LustreError: 165428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4900.631659] LustreError: 164681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.41@tcp arrived at 1786805435 with bad export cookie 14964121663673544757 [ 4900.647327] LustreError: 164681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4902.890768] LustreError: 164699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.951712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.004417] Lustre: 164009:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786805427/real 1786805427] req@000000002b8783af x1873601210902144/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786805443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4908.026146] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4913.120128] LustreError: 164699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.146308] LustreError: 164699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4918.243116] LustreError: 164008:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007d987459 x1873601210907776/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4918.551731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.614341] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4922.670981] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 4923.885915] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4923.896683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4924.750105] Lustre: lustre-MDT0000: Denying connection for new client f889b9cf-0f24-4f73-8f3b-73588ba66888 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4934.117120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.123141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.141701] Lustre: Skipped 1 previous similar message [ 4934.173931] Lustre: Skipped 1 previous similar message [ 4939.241124] LustreError: 166521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4939.269452] LustreError: 166521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4939.388956] Lustre: server umount lustre-OST0000 complete [ 4943.394156] LustreError: 168429:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4943.404794] Lustre: 167776:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4943.412349] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4943.444426] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4943.455367] LustreError: 167776:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c52b076c x1873601210921344/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4943.467817] LustreError: 167776:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4943.474868] LustreError: 167776:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4943.481673] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 4943.789075] Lustre: server umount lustre-MDT0000 complete [ 4944.354834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.075484] LustreError: 164679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805482 with bad export cookie 14964121663673544855 [ 4947.086067] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.095857] LustreError: 164679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4949.448661] Lustre: server umount lustre-MDT0001 complete [ 4958.752740] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 4961.004559] Key type lgssc unregistered [ 4961.216684] LNet: 169180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4961.223702] LNetError: 169180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4961.242797] LNet: Removed LNI 192.168.203.141@tcp [ 4961.896165] Key type .llcrypt unregistered [ 4961.898404] Key type ._llcrypt unregistered [ 4981.556049] Key type ._llcrypt registered [ 4981.558538] Key type .llcrypt registered [ 4981.647728] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 4982.452680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4982.552135] alg: No test for adler32 (adler32-zlib) [ 4983.616691] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4983.834413] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 4985.473191] Key type lgssc registered [ 4986.607614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4993.767249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4993.801677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.223183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.426365] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5005.802811] LustreError: 170572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.825244] LustreError: 170572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5007.721946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.189946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5012.412606] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5018.640429] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5023.024747] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5028.699508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5028.947278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.355906] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5040.242404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5041.622715] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5048.800645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5048.809278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.824136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5050.853643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5050.854629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5050.855896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.882611] Lustre: Skipped 1 previous similar message [ 5053.752401] Lustre: server umount lustre-OST0000 complete [ 5059.552902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5059.561869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.580446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5063.552326] Lustre: server umount lustre-MDT0000 complete [ 5066.210378] LustreError: 170567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5066.224795] LustreError: 170567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5066.719097] LustreError: 170554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805601 with bad export cookie 8578614964913395063 [ 5066.731884] LustreError: 170554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5066.735223] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.040713] Lustre: server umount lustre-MDT0001 complete [ 5077.691453] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5080.131131] Key type lgssc unregistered [ 5080.403863] LNet: 174103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5080.410200] LNetError: 174103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5080.422809] LNet: Removed LNI 192.168.203.141@tcp [ 5081.218660] Key type .llcrypt unregistered [ 5081.223696] Key type ._llcrypt unregistered [ 5106.730344] Key type ._llcrypt registered [ 5106.732506] Key type .llcrypt registered [ 5106.816744] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 5107.882833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5107.912181] alg: No test for adler32 (adler32-zlib) [ 5108.920668] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5109.201627] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 5110.895684] Key type lgssc registered [ 5111.949936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5118.616087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5118.629931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.935717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.421976] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5129.831482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.113638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5133.316925] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5142.675503] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5150.810399] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5158.200186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5158.698196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.889441] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5165.872575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5172.756795] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5185.505611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.539360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.545288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5188.470489] Lustre: server umount lustre-OST0000 complete [ 5195.746422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.756101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5195.770189] Lustre: Skipped 2 previous similar messages [ 5195.796448] Lustre: Skipped 2 previous similar messages [ 5200.867820] LustreError: 175778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.881937] LustreError: 175778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5200.938978] Lustre: server umount lustre-MDT0000 complete [ 5204.973993] LustreError: 177127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.016840] LustreError: 177127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5206.148338] LustreError: 175765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805741 with bad export cookie 5756044139059571385 [ 5206.161866] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5206.648685] Lustre: server umount lustre-MDT0001 complete [ 5216.496342] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5219.021706] Key type lgssc unregistered [ 5219.335029] LNet: 179316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5219.353164] LNetError: 179316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5219.370272] LNet: Removed LNI 192.168.203.141@tcp [ 5220.155201] Key type .llcrypt unregistered [ 5220.157257] Key type ._llcrypt unregistered [ 5238.278548] Key type ._llcrypt registered [ 5238.280658] Key type .llcrypt registered [ 5238.365772] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 5239.121470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5239.163517] alg: No test for adler32 (adler32-zlib) [ 5240.140596] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5240.305842] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 5241.935199] Key type lgssc registered [ 5242.808470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5251.056284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5251.072652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.282500] Lustre: *** cfs_fail_loc=135, val=0*** [ 5252.346028] LustreError: 180820:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5252.349238] LustreError: 180820:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.141@tcp: cfg command failed: rc = -2 [ 5252.355544] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5252.355544] [ 5252.360442] LustreError: MGC192.168.203.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5252.371069] LustreError: 180780:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5252.376780] LustreError: 180780:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5252.382321] LustreError: 180780:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5252.386382] LustreError: 180780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5252.433934] Lustre: server umount lustre-MDT0000 complete [ 5252.435793] LustreError: 180780:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5256.849244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.135522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.482769] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5266.836587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.260396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5271.451514] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5279.259390] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5284.796778] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5287.904630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5287.914230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.943503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5288.942305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5288.961069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.067684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5292.081065] Lustre: Skipped 2 previous similar messages [ 5293.652642] Lustre: server umount lustre-MDT0000 complete [ 5297.120512] LustreError: 181191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.146970] LustreError: 181191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5297.877036] LustreError: 181925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786805833 with bad export cookie 8875846954601843461 [ 5297.903123] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5298.642820] Lustre: server umount lustre-MDT0001 complete [ 5309.456707] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5312.208696] Key type lgssc unregistered [ 5312.444931] LNet: 183510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5312.452894] LNetError: 183510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5312.468739] LNet: Removed LNI 192.168.203.141@tcp [ 5313.395529] Key type .llcrypt unregistered [ 5313.400076] Key type ._llcrypt unregistered [ 5335.479283] Key type ._llcrypt registered [ 5335.481590] Key type .llcrypt registered [ 5335.606599] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5335.985501] Key type .llcrypt unregistered [ 5335.993547] Key type ._llcrypt unregistered [ 5351.494556] Key type ._llcrypt registered [ 5351.497051] Key type .llcrypt registered [ 5351.602540] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 5352.857750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.961274] alg: No test for adler32 (adler32-zlib) [ 5354.019866] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5354.321373] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 5356.007492] Key type lgssc registered [ 5356.996525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5363.650555] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5363.673566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5379.167371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5384.287292] LustreError: 185640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.141@tcp: failed processing log, type 4: rc = -110 [ 5410.911199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.194672] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5428.489503] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.585131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.209198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.315868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5439.829852] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5440.481514] LustreError: 185665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5440.511416] LustreError: 185665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5445.608241] LustreError: 185666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.643886] LustreError: 185666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5448.676612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.940475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5453.683695] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5462.357400] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5468.366474] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5470.996270] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5479.908071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.930498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5479.935371] Lustre: Skipped 1 previous similar message [ 5485.036428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5485.044116] Lustre: Skipped 1 previous similar message [ 5485.522869] Lustre: server umount lustre-OST0000 complete [ 5490.147296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.147853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5490.172977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.182114] Lustre: Skipped 1 previous similar message [ 5495.264704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5495.273324] Lustre: Skipped 1 previous similar message [ 5495.632522] Lustre: server umount lustre-MDT0000 complete [ 5499.101148] LustreError: 185647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806034 with bad export cookie 129597009630831780 [ 5499.114065] LustreError: 185647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5499.114800] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5499.463711] Lustre: server umount lustre-MDT0001 complete [ 5510.604557] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5513.277252] Key type lgssc unregistered [ 5513.526149] LNet: 189558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5513.537527] LNetError: 189558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.551216] LNet: Removed LNI 192.168.203.141@tcp [ 5514.332195] Key type .llcrypt unregistered [ 5514.338178] Key type ._llcrypt unregistered [ 5538.169319] Key type ._llcrypt registered [ 5538.171634] Key type .llcrypt registered [ 5538.259865] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 5539.701962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5539.749498] alg: No test for adler32 (adler32-zlib) [ 5540.798598] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5541.077526] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 5542.761190] Key type lgssc registered [ 5543.677835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5550.464279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5550.485669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.136732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.660602] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5562.363785] LustreError: 190983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.407693] LustreError: 190983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5565.839913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.411309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5571.228181] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5578.071422] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5583.813094] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5590.282091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5590.726892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.855414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5597.080933] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5603.418954] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5609.545143] Lustre: Failing over lustre-MDT0000 [ 5609.854552] Lustre: server umount lustre-MDT0000 complete [ 5611.476326] LustreError: 190978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.514951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.525041] LustreError: 190984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.536375] Lustre: Skipped 2 previous similar messages [ 5612.565580] LustreError: 190984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5616.562367] LustreError: 190978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.579059] LustreError: 190978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5621.683564] LustreError: 190980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.715039] LustreError: 190980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5627.873583] Lustre: 190296:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786806147/real 1786806147] req@0000000022b01db8 x1873601967973376/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786806163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5627.904634] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.928674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.732424] LustreError: 191985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.751393] LustreError: 191985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5638.128631] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd45f27f6ef91e00e [ 5638.140630] Lustre: MGC192.168.203.141@tcp: Connection restored to 0@lo (at 0@lo) [ 5638.535671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.593837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5643.185661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5643.556763] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5643.754034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5643.786045] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5643.845795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5648.066584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5657.352602] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5668.838694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5668.845199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.854978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.148525] Lustre: server umount lustre-OST0000 complete [ 5674.467442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.470361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5674.485547] Lustre: Skipped 1 previous similar message [ 5674.486424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5674.506748] Lustre: Skipped 3 previous similar messages [ 5679.586061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5679.589694] Lustre: Skipped 1 previous similar message [ 5684.705178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5684.710870] Lustre: Skipped 1 previous similar message [ 5688.287850] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5688.425512] Lustre: server umount lustre-MDT0000 complete [ 5689.827071] LustreError: 190978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.841859] LustreError: 190978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5691.642174] LustreError: 190964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806226 with bad export cookie 15302994000362987534 [ 5691.650589] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5691.654289] LustreError: 190964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5692.006890] Lustre: server umount lustre-MDT0001 complete [ 5702.805963] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5705.358051] Key type lgssc unregistered [ 5705.625933] LNet: 195925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5705.636255] LNetError: 195925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5705.651870] LNet: Removed LNI 192.168.203.141@tcp [ 5706.385264] Key type .llcrypt unregistered [ 5706.386880] Key type ._llcrypt unregistered [ 5730.385202] Key type ._llcrypt registered [ 5730.388385] Key type .llcrypt registered [ 5730.493604] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 5731.661576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5731.850289] alg: No test for adler32 (adler32-zlib) [ 5732.830711] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5733.031095] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 5734.730132] Key type lgssc registered [ 5736.283165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5745.164173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5745.183379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.521425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5749.875991] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5755.878979] LustreError: 197354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.900406] LustreError: 197354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5757.504146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.757909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5761.378193] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5766.967901] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5771.588324] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5778.289985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.636878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.507420] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5790.906783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5792.424702] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5796.452961] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5800.134618] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5819.492989] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5823.975278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.980945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.988474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5827.046147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5827.046466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.047367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5829.088457] Lustre: server umount lustre-OST0000 complete [ 5834.721283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5834.737108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.760409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.767167] Lustre: Skipped 1 previous similar message [ 5837.284081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5837.288290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.009259] Lustre: server umount lustre-MDT0000 complete [ 5842.410917] LustreError: 197351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.433063] LustreError: 197351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5843.182813] LustreError: 199888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806378 with bad export cookie 13901484592849500611 [ 5843.209766] LustreError: 199888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5843.211808] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.558276] Lustre: server umount lustre-MDT0001 complete [ 5856.202822] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 5859.715913] Key type lgssc unregistered [ 5860.238857] LNet: 201219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5860.259940] LNetError: 201219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5861.294749] LNet: Removed LNI 192.168.203.141@tcp [ 5862.555873] Key type .llcrypt unregistered [ 5862.560275] Key type ._llcrypt unregistered [ 5881.935814] Key type ._llcrypt registered [ 5881.940748] Key type .llcrypt registered [ 5882.062839] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 5883.152458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5883.199218] alg: No test for adler32 (adler32-zlib) [ 5884.374459] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5884.636765] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 5886.319112] Key type lgssc registered [ 5887.520927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5895.449559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5895.479887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.952149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.480931] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5907.434051] LustreError: 202488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.464436] LustreError: 202488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5909.243636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.508894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5913.414244] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5920.159571] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.654291] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5931.532471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5931.902365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.448506] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 5942.129161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5943.233586] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5947.731257] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5957.459230] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5965.752793] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5972.904678] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5983.205554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5983.211613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.220739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.242114] Lustre: Skipped 1 previous similar message [ 5987.394846] Lustre: server umount lustre-OST0000 complete [ 5991.393947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.401964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.418470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5991.433664] Lustre: Skipped 1 previous similar message [ 5993.440488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.442935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.560817] LustreError: 202488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.590966] LustreError: 202488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5999.308404] Lustre: server umount lustre-MDT0000 complete [ 6002.482716] LustreError: 202469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806537 with bad export cookie 5801934796817238917 [ 6002.490473] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.498028] LustreError: 202469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6002.806369] Lustre: server umount lustre-MDT0001 complete [ 6012.388347] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6014.748040] Key type lgssc unregistered [ 6014.987899] LNet: 207411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6014.994873] LNetError: 207411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6015.009821] LNet: Removed LNI 192.168.203.141@tcp [ 6015.720978] Key type .llcrypt unregistered [ 6015.722577] Key type ._llcrypt unregistered [ 6033.897066] Key type ._llcrypt registered [ 6033.899393] Key type .llcrypt registered [ 6034.001411] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6034.662484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6034.727249] alg: No test for adler32 (adler32-zlib) [ 6035.793912] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6036.049400] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6037.761978] Key type lgssc registered [ 6038.779840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6045.851129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6045.873297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.274542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.453789] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6057.454356] LustreError: 208803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.470600] LustreError: 208803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6058.891348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6059.304592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6063.017992] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6070.827853] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6076.230848] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6082.897934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6083.405861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.909843] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6092.732023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6097.409322] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6113.251084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.252210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6113.256352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6113.279284] Lustre: Skipped 1 previous similar message [ 6117.484682] Lustre: server umount lustre-OST0000 complete [ 6123.490869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.493971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.507642] Lustre: Skipped 1 previous similar message [ 6123.513246] Lustre: Skipped 1 previous similar message [ 6128.146156] Lustre: server umount lustre-MDT0000 complete [ 6128.609915] LustreError: 208803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.644423] LustreError: 208803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6132.938606] LustreError: 210650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806668 with bad export cookie 7694229103071043405 [ 6132.940443] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6132.963573] LustreError: 210650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6133.474402] Lustre: server umount lustre-MDT0001 complete [ 6149.856869] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6152.382821] Key type lgssc unregistered [ 6152.690451] LNet: 212489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6152.696921] LNetError: 212489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6152.712455] LNet: Removed LNI 192.168.203.141@tcp [ 6153.674538] Key type .llcrypt unregistered [ 6153.678912] Key type ._llcrypt unregistered [ 6175.669809] Key type ._llcrypt registered [ 6175.673021] Key type .llcrypt registered [ 6175.914060] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6178.556543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6178.592975] alg: No test for adler32 (adler32-zlib) [ 6179.852351] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6180.203162] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6181.879691] Key type lgssc registered [ 6182.914624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6194.579521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6194.618949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.363378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.570718] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6205.953931] LustreError: 213755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.988551] LustreError: 213755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6211.048641] LustreError: 213756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.840899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.093552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.144474] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6222.526330] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6227.065909] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6233.410328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6233.830613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.231325] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6241.043578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6247.676789] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6261.729678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.731368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6261.755853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6261.780167] Lustre: Skipped 1 previous similar message [ 6266.564373] Lustre: server umount lustre-OST0000 complete [ 6271.977328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.984937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6272.001271] Lustre: Skipped 2 previous similar messages [ 6277.091431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6277.098152] Lustre: Skipped 1 previous similar message [ 6282.209133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.225871] Lustre: Skipped 1 previous similar message [ 6284.255595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6284.420402] Lustre: server umount lustre-MDT0000 complete [ 6287.333270] LustreError: 213756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.360181] LustreError: 213756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6288.817843] LustreError: 213737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806823 with bad export cookie 16932421738414414446 [ 6288.825857] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.838716] LustreError: 213737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6289.318640] Lustre: server umount lustre-MDT0001 complete [ 6300.493319] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6302.929191] Key type lgssc unregistered [ 6303.187679] LNet: 217441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6303.192024] LNetError: 217441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6303.204219] LNet: Removed LNI 192.168.203.141@tcp [ 6304.103575] Key type .llcrypt unregistered [ 6304.105156] Key type ._llcrypt unregistered [ 6321.551579] Key type ._llcrypt registered [ 6321.559518] Key type .llcrypt registered [ 6321.689274] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6322.495370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6322.514335] alg: No test for adler32 (adler32-zlib) [ 6323.636107] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6323.903955] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6325.600871] Key type lgssc registered [ 6326.801276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6333.399659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6333.419471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.868496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6338.940763] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6345.186761] LustreError: 218707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.202615] LustreError: 218707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6346.343607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.583561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6350.839396] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6358.473925] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6364.241961] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6371.110165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6371.297294] LustreError: 220526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.554515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.884089] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6380.041252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6383.809204] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6392.614517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.694532] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6392.884930] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6398.520456] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6406.477934] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6408.090094] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6419.642576] Lustre: Permanently deactivating lustre-OST0001 [ 6419.654067] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6427.740942] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6433.960102] Lustre: Permanently reactivating lustre-OST0001 [ 6433.976228] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6433.981561] Lustre: Skipped 2 previous similar messages [ 6441.083669] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.114835] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6441.128697] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6441.146163] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6441.215588] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6446.562388] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6446.564798] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.582021] LustreError: Skipped 1 previous similar message [ 6446.588668] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6446.642331] Lustre: Skipped 2 previous similar messages [ 6447.541734] Lustre: server umount lustre-OST0001 complete [ 6448.616134] LustreError: 220529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.638974] LustreError: 220529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6453.728809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.735431] LustreError: 220529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.735741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.738744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.738751] Lustre: Skipped 1 previous similar message [ 6453.796098] LustreError: 220529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6458.852354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6458.853345] LustreError: 220526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.861347] Lustre: Skipped 2 previous similar messages [ 6458.877162] LustreError: 220526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6463.975271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.015253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6466.186686] Lustre: server umount lustre-OST0000 complete [ 6474.211575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.216181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6474.236853] Lustre: Skipped 2 previous similar messages [ 6474.244319] Lustre: Skipped 2 previous similar messages [ 6476.868048] Lustre: server umount lustre-MDT0000 complete [ 6479.339986] LustreError: 219434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.372040] LustreError: 219434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6481.582869] LustreError: 218688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807016 with bad export cookie 1334904659732076529 [ 6481.586754] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.594249] LustreError: 218688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6482.512201] Lustre: server umount lustre-MDT0001 complete [ 6492.155153] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6494.483113] Key type lgssc unregistered [ 6494.720731] LNet: 223753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6494.727968] LNetError: 223753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6494.743227] LNet: Removed LNI 192.168.203.141@tcp [ 6495.405206] Key type .llcrypt unregistered [ 6495.407918] Key type ._llcrypt unregistered [ 6528.395701] Key type ._llcrypt registered [ 6528.398059] Key type .llcrypt registered [ 6528.525394] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6529.477443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6529.630351] alg: No test for adler32 (adler32-zlib) [ 6530.695734] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6530.936772] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6532.575173] Key type lgssc registered [ 6533.659840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6540.287913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6540.299312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.516786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6541.531554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6541.654396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.236411] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6552.148081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.204292] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6552.226334] Lustre: 225952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.398758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6556.150479] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6563.233932] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6568.054350] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6573.134482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.300961] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6573.498231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6578.744068] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6583.430105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6586.038251] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.602634] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6593.988907] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6601.560721] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6609.079531] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6624.832886] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6624.841317] Lustre: Skipped 1 previous similar message [ 6643.472358] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6643.481110] Lustre: Skipped 2 previous similar messages [ 6679.117250] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6679.124727] Lustre: Skipped 3 previous similar messages [ 6743.334429] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6743.346857] Lustre: Skipped 7 previous similar messages [ 6757.865974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.870793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6757.878225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6762.465387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.469190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.481863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6763.818407] Lustre: server umount lustre-OST0000 complete [ 6767.585509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6767.602120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.622073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.643570] Lustre: Skipped 1 previous similar message [ 6773.219118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6773.228881] Lustre: Skipped 3 previous similar messages [ 6773.452985] Lustre: server umount lustre-MDT0000 complete [ 6777.379486] LustreError: 228690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807312 with bad export cookie 4687250289902345154 [ 6777.388402] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.400184] LustreError: 228690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6777.828596] Lustre: server umount lustre-MDT0001 complete [ 6788.507073] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6791.138526] Key type lgssc unregistered [ 6791.450238] LNet: 230782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6791.459533] LNetError: 230782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6791.496146] LNet: Removed LNI 192.168.203.141@tcp [ 6792.185514] Key type .llcrypt unregistered [ 6792.188549] Key type ._llcrypt unregistered [ 6812.852030] Key type ._llcrypt registered [ 6812.854807] Key type .llcrypt registered [ 6812.949621] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6813.683306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6813.730391] alg: No test for adler32 (adler32-zlib) [ 6814.802428] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6815.042168] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6816.679475] Key type lgssc registered [ 6817.992593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6825.555287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6825.576478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.986251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.088259] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6837.218550] LustreError: 232176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.229877] LustreError: 232176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6838.652874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.005083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6843.104118] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6850.207872] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6855.274220] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6860.968026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6861.215893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.912671] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug -1 all [ 6872.523255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6873.959207] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.911486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.919192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6887.933061] Lustre: Skipped 1 previous similar message [ 6887.952477] Lustre: Skipped 1 previous similar message [ 6889.432347] Lustre: server umount lustre-OST0000 complete [ 6893.025079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.036620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6893.040091] Lustre: Skipped 1 previous similar message [ 6898.145885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.264464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.272096] Lustre: Skipped 2 previous similar messages [ 6907.359418] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6907.560531] Lustre: server umount lustre-MDT0000 complete [ 6908.393862] LustreError: 232170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.410472] LustreError: 232170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6910.899947] LustreError: 232909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807446 with bad export cookie 3324109399976161544 [ 6910.902425] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6910.911646] LustreError: 232909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6911.120384] Lustre: server umount lustre-MDT0001 complete [ 6920.183679] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6922.817298] Key type lgssc unregistered [ 6923.093970] LNet: 235864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6923.100869] LNetError: 235864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6923.117677] LNet: Removed LNI 192.168.203.141@tcp [ 6923.883448] Key type .llcrypt unregistered [ 6923.885470] Key type ._llcrypt unregistered [ 6943.611480] Key type ._llcrypt registered [ 6943.613830] Key type .llcrypt registered [ 6943.729813] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 6944.562881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6944.643157] alg: No test for adler32 (adler32-zlib) [ 6945.664718] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 6945.829979] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 6947.479565] Key type lgssc registered [ 6948.380578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7219.070419] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 7221.979879] Key type lgssc unregistered [ 7222.204184] LNet: 237889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7222.212708] LNetError: 237889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7222.231963] LNet: Removed LNI 192.168.203.141@tcp [ 7222.813590] Key type .llcrypt unregistered [ 7222.815414] Key type ._llcrypt unregistered [ 7255.783319] Key type ._llcrypt registered [ 7255.785502] Key type .llcrypt registered [ 7255.919262] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 7273.574551] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 7275.001789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7275.036078] alg: No test for adler32 (adler32-zlib) [ 7276.031943] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 7276.233528] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 7277.887214] Key type lgssc registered [ 7279.060763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7309.184815] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7029 sec ========== 11:30:43 (1786807843) [ 7311.109555] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:30:45 (1786807845) === [ 7314.055362] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:30:48 (1786807848) === [ 7351.426099] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 7353.836336] Key type lgssc unregistered [ 7354.108164] LNet: 243127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7354.113629] LNetError: 243127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7354.125915] LNet: Removed LNI 192.168.203.141@tcp [ 7354.833746] Key type .llcrypt unregistered [ 7354.836686] Key type ._llcrypt unregistered