[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478339089 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002403] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.006012] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.022284] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027182] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] *** VALIDATE tmpfs *** [ 0.031472] *** VALIDATE proc *** [ 0.032251] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036078] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041896] debug: unmapping init [mem 0xffffffff84259000-0xffffffff84260fff] [ 0.043909] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044707] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052311] rcu: Hierarchical SRCU implementation. [ 0.054472] smp: Bringing up secondary CPUs ... [ 0.055602] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.059444] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141591] node 0 deferred pages initialised in 76ms [ 0.143248] devtmpfs: initialized [ 0.146210] x86/mm: Memory block size: 128MB [ 0.150313] gcov: version magic: 0x41383552 [ 0.154294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160121] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163330] pinctrl core: initialized pinctrl subsystem [ 0.165199] [ 0.165652] ************************************************************* [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ** ** [ 0.173012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176013] ** ** [ 0.178009] ** This means that this kernel is built to expose internal ** [ 0.181012] ** IOMMU data structures, which may compromise security on ** [ 0.183010] ** your system. ** [ 0.185013] ** ** [ 0.187014] ** If you see this message and you are not debugging the ** [ 0.189012] ** kernel, report this immediately to your vendor! ** [ 0.192023] ** ** [ 0.194012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197028] ************************************************************* [ 0.200634] NET: Registered protocol family 16 [ 0.202362] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215027] cpuidle: using governor menu [ 0.216632] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219460] PCI: Using configuration type 1 for base access [ 0.222127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.237059] cryptd: max_cpu_qlen set to 1000 [ 0.240113] ACPI: Added _OSI(Module Device) [ 0.241012] ACPI: Added _OSI(Processor Device) [ 0.243014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250735] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256477] ACPI: Interpreter enabled [ 0.258067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260014] ACPI: Using IOAPIC for interrupt routing [ 0.262104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276949] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290424] acpiphp: Slot [2] registered [ 0.292098] acpiphp: Slot [5] registered [ 0.293119] acpiphp: Slot [6] registered [ 0.295120] acpiphp: Slot [7] registered [ 0.296114] acpiphp: Slot [8] registered [ 0.298114] acpiphp: Slot [9] registered [ 0.299117] acpiphp: Slot [10] registered [ 0.301125] acpiphp: Slot [3] registered [ 0.302123] acpiphp: Slot [4] registered [ 0.304108] acpiphp: Slot [11] registered [ 0.305102] acpiphp: Slot [12] registered [ 0.307121] acpiphp: Slot [13] registered [ 0.309113] acpiphp: Slot [14] registered [ 0.310129] acpiphp: Slot [15] registered [ 0.312115] acpiphp: Slot [16] registered [ 0.314112] acpiphp: Slot [17] registered [ 0.316083] acpiphp: Slot [18] registered [ 0.317117] acpiphp: Slot [19] registered [ 0.319148] acpiphp: Slot [20] registered [ 0.321123] acpiphp: Slot [21] registered [ 0.322082] acpiphp: Slot [22] registered [ 0.324137] acpiphp: Slot [23] registered [ 0.325108] acpiphp: Slot [24] registered [ 0.327187] acpiphp: Slot [25] registered [ 0.329102] acpiphp: Slot [26] registered [ 0.330108] acpiphp: Slot [27] registered [ 0.332106] acpiphp: Slot [28] registered [ 0.333083] acpiphp: Slot [29] registered [ 0.335120] acpiphp: Slot [30] registered [ 0.336102] acpiphp: Slot [31] registered [ 0.338086] PCI host bridge to bus 0000:00 [ 0.340025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.348024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.351025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355332] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359040] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362211] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.371826] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.379018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.382018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388580] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390839] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397876] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.404014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.416972] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421989] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.427624] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464730] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.500019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510685] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.523017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.540015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550863] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.592454] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.607015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.622013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.630272] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.637018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.648072] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682613] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.685287] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.688535] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691415] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.693220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698104] iommu: Default domain type: Passthrough [ 0.700376] SCSI subsystem initialized [ 0.702194] ACPI: bus type USB registered [ 0.703113] usbcore: registered new interface driver usbfs [ 0.705078] usbcore: registered new interface driver hub [ 0.707075] usbcore: registered new device driver usb [ 0.709167] pps_core: LinuxPPS API ver. 1 registered [ 0.711010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.714059] PTP clock support registered [ 0.716052] EDAC MC: Ver: 3.0.0 [ 0.718172] PCI: Using ACPI for IRQ routing [ 0.719748] NetLabel: Initializing [ 0.721010] NetLabel: domain hash size = 128 [ 0.723009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.725081] NetLabel: unlabeled traffic allowed by default [ 0.728136] vgaarb: loaded [ 0.729270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.730012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.735620] clocksource: Switched to clocksource kvm-clock [ 0.842427] VFS: Disk quotas dquot_6.6.0 [ 0.844028] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.846535] *** VALIDATE ramfs *** [ 0.847972] *** VALIDATE hugetlbfs *** [ 0.849507] pnp: PnP ACPI init [ 0.851986] pnp: PnP ACPI: found 6 devices [ 0.875163] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.879114] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.881024] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.883416] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.885809] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.889680] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.892379] NET: Registered protocol family 2 [ 0.894678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899207] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.903325] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.909207] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.913347] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917268] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.920613] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923704] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927040] NET: Registered protocol family 1 [ 0.929463] RPC: Registered named UNIX socket transport module. [ 0.932103] RPC: Registered udp transport module. [ 0.934259] RPC: Registered tcp transport module. [ 0.936382] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939267] NET: Registered protocol family 44 [ 0.941330] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.943889] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.946400] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.948434] PCI: CLS 0 bytes, default 64 [ 0.950150] Unpacking initramfs... [ 2.352445] debug: unmapping init [mem 0xffff8ef1fcc54000-0xffff8ef1fffbffff] [ 2.356450] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.359117] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.362307] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.861874] Initialise system trusted keyrings [ 2.863936] Key type blacklist registered [ 2.865948] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.876880] zbud: loaded [ 2.879970] *** VALIDATE nfs *** [ 2.881471] *** VALIDATE nfs4 *** [ 2.883268] pstore: using deflate compression [ 2.888514] Platform Keyring initialized [ 2.986883] NET: Registered protocol family 38 [ 2.989299] Key type asymmetric registered [ 2.991249] Asymmetric key parser 'x509' registered [ 2.993416] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.997177] io scheduler mq-deadline registered [ 2.999041] io scheduler kyber registered [ 3.001172] io scheduler bfq registered [ 3.003132] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.006553] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010138] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.013163] ACPI: Power Button [PWRF] [ 3.019050] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025575] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.040180] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.049126] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.062129] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091126] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.121804] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.127993] Non-volatile memory driver v1.3 [ 3.129788] Linux agpgart interface v0.103 [ 3.164570] virtio_blk virtio1: [vda] 146160 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.169112] vda: detected capacity change from 0 to 74833920 [ 3.183268] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.186558] vdb: detected capacity change from 0 to 1073741824 [ 3.202162] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205588] vdc: detected capacity change from 0 to 2621440000 [ 3.224785] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228819] vdd: detected capacity change from 0 to 2621440000 [ 3.245965] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249100] vde: detected capacity change from 0 to 4294967296 [ 3.264401] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.267712] vdf: detected capacity change from 0 to 4294967296 [ 3.274450] libphy: Fixed MDIO Bus: probed [ 3.284913] usbcore: registered new interface driver usbserial_generic [ 3.287765] usbserial: USB Serial support registered for generic [ 3.290661] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.295498] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.297461] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.300398] mousedev: PS/2 mouse device common for all mice [ 3.303774] rtc_cmos 00:05: RTC can wake from S4 [ 3.307295] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.310053] rtc_cmos 00:05: registered as rtc0 [ 3.313506] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.316038] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317823] intel_pstate: CPU model not supported [ 3.324065] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.329107] hid: raw HID events driver (C) Jiri Kosina [ 3.331810] usbcore: registered new interface driver usbhid [ 3.334276] usbhid: USB HID core driver [ 3.336153] drop_monitor: Initializing network drop monitor service [ 3.338908] Initializing XFRM netlink socket [ 3.341525] NET: Registered protocol family 10 [ 3.348420] Segment Routing with IPv6 [ 3.350297] NET: Registered protocol family 17 [ 3.352752] mpls_gso: MPLS GSO support [ 3.360931] RAS: Correctable Errors collector initialized. [ 3.363498] AVX version of gcm_enc/dec engaged. [ 3.365399] AES CTR mode by8 optimization enabled [ 3.458656] sched_clock: Marking stable (3458634076, 0)->(4410141328, -951507252) [ 3.461899] registered taskstats version 1 [ 3.463535] Loading compiled-in X.509 certificates [ 3.465590] zswap: loaded using pool lzo/zbud [ 3.491205] Key type big_key registered [ 3.504447] Key type encrypted registered [ 3.506301] ima: No TPM chip found, activating TPM-bypass! [ 3.508466] ima: Allocated hash algorithm: sha1 [ 3.509767] ima: No architecture policies found [ 3.511137] evm: Initialising EVM extended attributes: [ 3.513282] evm: security.selinux [ 3.514502] evm: security.ima [ 3.515844] evm: security.capability [ 3.517345] evm: HMAC attrs: 0x1 [ 3.519923] rtc_cmos 00:05: setting system clock to 2026-08-19 11:01:00 UTC (1787137260) [ 3.526354] debug: unmapping init [mem 0xffffffff85203000-0xffffffff853fffff] [ 3.529402] debug: unmapping init [mem 0xffffffff83f82000-0xffffffff84258fff] [ 3.537161] Write protecting the kernel read-only data: 28672k [ 3.541240] debug: unmapping init [mem 0xffffffff82603000-0xffffffff827fffff] [ 3.544049] debug: unmapping init [mem 0xffffffff82f14000-0xffffffff82ffffff] [ 3.582057] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.589809] systemd[1]: Detected virtualization kvm. [ 3.592423] systemd[1]: Detected architecture x86-64. [ 3.594844] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.618365] systemd[1]: No hostname configured. [ 3.620350] systemd[1]: Set hostname to . [ 3.623180] random: systemd: uninitialized urandom read (16 bytes read) [ 3.626370] systemd[1]: Initializing machine ID from random generator. [ 3.672108] random: ln: uninitialized urandom read (6 bytes read) [ 3.768279] random: systemd: uninitialized urandom read (16 bytes read) [ 3.771300] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.777964] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.782218] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.347403] device-mapper: uevent: version 1.0.3 [ 4.349994] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.005968] virtio_net virtio0 ens2: renamed from eth0 [ 5.030851] random: fast init done [ 5.075546] scsi host0: ata_piix [ 5.105892] scsi host1: ata_piix [ 5.127782] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.130898] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.597419] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.022759] random: crng init done [ 10.024535] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.269994] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.332625] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.580625] SELinux: Disabled at runtime. [ 11.642604] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.651812] systemd[1]: Detected virtualization kvm. [ 11.653865] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.169855] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.172753] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.177138] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.183252] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.185797] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.192436] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.198190] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ 12.322275] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.591866] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.874483] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.892563] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.026209] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.043856] EDAC sbridge: Ver: 1.1.2 [ 14.554515] Key type dns_resolver registered [ 14.861955] NFS: Registering the id_resolver key type [ 14.864266] Key type id_resolver registered [ 14.866072] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg312-server login: [ 40.342376] libcfs: loading out-of-tree module taints kernel. [ 40.370188] Key type ._llcrypt registered [ 40.372164] Key type .llcrypt registered [ 40.427569] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 51.378198] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 53.146868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.180684] alg: No test for adler32 (adler32-zlib) [ 54.694307] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 55.907254] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 57.863209] Key type lgssc registered [ 59.602831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.672648] hrtimer: interrupt took 5114667 ns [ 75.261309] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.340713] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 126.756353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 126.783210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.990723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 128.025558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 128.140702] Lustre: lustre-MDT0000: new disk, initializing [ 128.197372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 128.227776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 131.623500] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.549821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.680723] Lustre: 6497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 143.708549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 143.711980] Lustre: Skipped 1 previous similar message [ 143.796731] Lustre: lustre-MDT0001: new disk, initializing [ 143.907094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 143.958022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 143.980628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.186882] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.085684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 163.294713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.769786] Lustre: lustre-OST0000: new disk, initializing [ 163.777039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 163.784496] Lustre: 8434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.837285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.745904] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.060287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.079263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.207132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 184.904916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.138782] Lustre: lustre-OST0001: new disk, initializing [ 185.151190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 185.165123] Lustre: 9505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.257700] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.943449] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.619806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 194.651052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 194.752339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 204.585802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 213.618349] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 219.347860] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing check_logdir /tmp/testlogs/ [ 225.415946] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing yml_node [ 230.284989] Lustre: DEBUG MARKER: Client: 2.17.57.8 [ 233.104020] Lustre: DEBUG MARKER: MDS: 2.17.57.8 [ 235.518688] Lustre: DEBUG MARKER: OSS: 2.17.57.8 [ 237.379821] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 07:04:53 EDT 2026 [ 254.078636] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 255.655372] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 271.337796] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.343151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 271.352528] Lustre: Skipped 3 previous similar messages [ 271.372192] Lustre: Skipped 3 previous similar messages [ 276.450755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 276.462134] Lustre: Skipped 3 previous similar messages [ 277.420425] Lustre: server umount lustre-MDT0000 complete [ 280.962855] LustreError: 7453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137537 with bad export cookie 6783729057130885511 [ 280.968689] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 280.991436] LustreError: 7453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 281.570552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.579219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 281.590374] Lustre: Skipped 1 previous similar message [ 281.615376] Lustre: Skipped 1 previous similar message [ 285.600635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.609303] Lustre: Skipped 1 previous similar message [ 290.784206] LustreError: 6503:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.785813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.805342] LustreError: 6503:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 6 previous similar messages [ 290.824263] Lustre: Skipped 1 previous similar message [ 291.808204] LustreError: 12246:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.827990] LustreError: 12246:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 295.391348] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 295.683137] Lustre: server umount lustre-MDT0001 complete [ 299.811289] Lustre: server umount lustre-OST0000 complete [ 303.402088] Lustre: server umount lustre-OST0001 complete [ 310.480380] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 321.926845] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 359.077617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.390922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.421513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.497984] Lustre: lustre-MDT0000: new disk, initializing [ 359.600143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.633113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 364.064311] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 374.255727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.398538] Lustre: 15059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.506799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.516770] Lustre: Skipped 1 previous similar message [ 374.611689] Lustre: lustre-MDT0001: new disk, initializing [ 374.735889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.806158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.813521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.079506] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 388.260448] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.138936] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 400.987752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 401.345471] Lustre: lustre-OST0000: new disk, initializing [ 401.352546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 401.361751] Lustre: 17028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 401.478636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.247260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 403.260807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 403.417942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 407.602920] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 417.169217] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 421.487714] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 421.662604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 425.837166] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 426.054300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 429.026360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 429.038804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.051569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 429.057546] Lustre: Skipped 3 previous similar messages [ 442.335661] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.516870] Lustre: server umount lustre-OST0000 complete [ 449.515056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.533116] Lustre: Skipped 1 previous similar message [ 449.538432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.543731] Lustre: Skipped 4 previous similar messages [ 452.869102] Lustre: server umount lustre-MDT0000 complete [ 454.625433] LustreError: 15066:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.655260] LustreError: 15066:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 457.294595] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137714 with bad export cookie 6783729057130887051 [ 457.296480] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.325067] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 457.697995] Lustre: server umount lustre-MDT0001 complete [ 461.306565] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:08:37 (1787137717) [ 468.345788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.865803] LustreError: 19292:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.928674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.081235] LustreError: 19293:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.222490] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 483.302622] LustreError: 19292:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.340946] LustreError: 19292:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 483.903794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.223355] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 496.039414] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.811258] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.206713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.970682] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 529.096686] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.629829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.630698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.635556] LustreError: Skipped 2 previous similar messages [ 539.639888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.639897] Lustre: Skipped 1 previous similar message [ 539.676526] Lustre: Skipped 2 previous similar messages [ 542.671331] Lustre: server umount lustre-OST0000 complete [ 549.858713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.625268] Lustre: server umount lustre-MDT0000 complete [ 554.976426] LustreError: 20021:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.994899] LustreError: 20021:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 556.015324] LustreError: 19273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137812 with bad export cookie 6783729057130887933 [ 556.023841] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.032339] LustreError: 19273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 556.503405] Lustre: server umount lustre-MDT0001 complete [ 567.273358] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 570.216420] Key type lgssc unregistered [ 570.561641] LNet: 22826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.567466] LNetError: 22826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.577679] LNet: Removed LNI 192.168.203.112@tcp [ 571.484178] Key type .llcrypt unregistered [ 571.487687] Key type ._llcrypt unregistered [ 594.054959] Key type ._llcrypt registered [ 594.057606] Key type .llcrypt registered [ 594.197599] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 595.727484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 595.758922] alg: No test for adler32 (adler32-zlib) [ 596.902789] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 597.172864] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 598.871366] Key type lgssc registered [ 600.165669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 607.939318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.973794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.596450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.108589] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 620.001115] LustreError: 24238:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.024433] LustreError: 24238:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 622.484262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.841848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 627.435925] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 634.800318] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 640.137634] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 646.675659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 647.163644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.086702] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 654.331468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 660.306717] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 677.389899] Lustre: server umount lustre-OST0000 complete [ 684.516423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 684.525463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.554458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.042762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.059598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.067686] Lustre: Skipped 1 previous similar message [ 690.149322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 690.172038] Lustre: Skipped 1 previous similar message [ 691.240269] Lustre: server umount lustre-MDT0000 complete [ 694.178544] Lustre: 23550:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787137935/real 1787137935] req@00000000f08a4a3b x1873949862425472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787137951 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 694.222319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.248564] LustreError: 24238:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.268798] LustreError: 24238:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 694.896882] LustreError: 24220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137951 with bad export cookie 11909626713829023581 [ 694.902893] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.910055] LustreError: 24220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 695.207377] Lustre: server umount lustre-MDT0001 complete [ 705.093526] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 707.554414] Key type lgssc unregistered [ 707.867756] LNet: 28218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.873515] LNetError: 28218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.885727] LNet: Removed LNI 192.168.203.112@tcp [ 708.668714] Key type .llcrypt unregistered [ 708.670803] Key type ._llcrypt unregistered [ 729.850703] Key type ._llcrypt registered [ 729.856568] Key type .llcrypt registered [ 729.946734] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 731.191262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.210576] alg: No test for adler32 (adler32-zlib) [ 732.369765] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 732.754236] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 734.455217] Key type lgssc registered [ 735.780351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 743.649573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 743.680416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.203526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.126561] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 755.697712] LustreError: 29611:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.732075] LustreError: 29611:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 759.213164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.595479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 764.108367] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 771.381195] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 776.634581] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 787.890264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.260536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.318582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 793.881367] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 801.546456] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 815.082977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 815.096044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.119189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 816.096478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 816.105240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.116302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.220437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.237371] Lustre: Skipped 2 previous similar messages [ 826.352362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.358857] Lustre: Skipped 1 previous similar message [ 827.359319] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 827.556075] Lustre: server umount lustre-OST0000 complete [ 831.456235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 831.457415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.467592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.477742] Lustre: Skipped 1 previous similar message [ 831.492367] Lustre: Skipped 1 previous similar message [ 837.688053] Lustre: server umount lustre-MDT0000 complete [ 841.688749] LustreError: 29594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138098 with bad export cookie 11998739832498469513 [ 841.698515] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.708860] LustreError: 29594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 842.017641] Lustre: server umount lustre-MDT0001 complete [ 853.097040] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 856.044769] Key type lgssc unregistered [ 856.285913] LNet: 33492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.294975] LNetError: 33492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.304884] LNet: Removed LNI 192.168.203.112@tcp [ 857.073921] Key type .llcrypt unregistered [ 857.080902] Key type ._llcrypt unregistered [ 879.925144] Key type ._llcrypt registered [ 879.927994] Key type .llcrypt registered [ 880.028503] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 881.336310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.396995] alg: No test for adler32 (adler32-zlib) [ 882.564241] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 882.881500] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 884.599166] Key type lgssc registered [ 885.617533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.599525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.628263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.081605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.439623] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 904.681371] LustreError: 34907:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.703440] LustreError: 34907:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 907.482571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.785221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.447508] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 919.156595] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 924.508854] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 931.253186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.553900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.584298] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 940.851108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 944.633891] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 953.313743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.318618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.327813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.830904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 953.842864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.856717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.400711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.407535] Lustre: Skipped 1 previous similar message [ 958.650735] Lustre: server umount lustre-OST0000 complete [ 964.064905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.077252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.094447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.624750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.996899] Lustre: server umount lustre-MDT0000 complete [ 971.752630] LustreError: 34903:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.769821] LustreError: 34903:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 972.907343] LustreError: 34888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138229 with bad export cookie 11819973524465265899 [ 972.914298] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.922620] LustreError: 34888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 973.396681] Lustre: server umount lustre-MDT0001 complete [ 984.648783] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 986.974407] Key type lgssc unregistered [ 987.394051] LNet: 38446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.413649] LNetError: 38446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.451672] LNet: Removed LNI 192.168.203.112@tcp [ 988.716356] Key type .llcrypt unregistered [ 988.720359] Key type ._llcrypt unregistered [ 1015.621761] Key type ._llcrypt registered [ 1015.624742] Key type .llcrypt registered [ 1015.699165] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1016.952167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.987392] alg: No test for adler32 (adler32-zlib) [ 1018.039888] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1018.247283] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1019.935313] Key type lgssc registered [ 1021.172855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.819491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.847063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.346414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.920577] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1041.894884] LustreError: 39857:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.923064] LustreError: 39857:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1044.491485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.847993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.587784] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1055.613833] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1060.896537] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1066.921466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.248833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.262240] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1076.543635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1079.854044] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1086.957673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.960225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.982061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.006307] Lustre: Skipped 1 previous similar message [ 1087.795205] Lustre: lustre-OST0000: Not available for connect from 192.168.203.12@tcp (stopping) [ 1092.065075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.078255] Lustre: Skipped 1 previous similar message [ 1097.185488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.193528] Lustre: Skipped 2 previous similar messages [ 1100.258570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1100.489320] Lustre: server umount lustre-OST0000 complete [ 1119.056945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.072350] Lustre: Skipped 1 previous similar message [ 1119.077606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.086826] Lustre: Skipped 2 previous similar messages [ 1125.475864] Lustre: server umount lustre-MDT0000 complete [ 1127.401624] LustreError: 40586:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.427820] LustreError: 40586:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 1129.340527] LustreError: 39839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138386 with bad export cookie 13854809782398722660 [ 1129.359973] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.784293] Lustre: server umount lustre-MDT0001 complete [ 1146.593437] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1149.300720] Key type lgssc unregistered [ 1149.697029] LNet: 43572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.705689] LNetError: 43572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.736880] LNet: Removed LNI 192.168.203.112@tcp [ 1150.863542] Key type .llcrypt unregistered [ 1150.869323] Key type ._llcrypt unregistered [ 1174.004388] Key type ._llcrypt registered [ 1174.007584] Key type .llcrypt registered [ 1174.122743] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1175.132435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.240350] alg: No test for adler32 (adler32-zlib) [ 1176.254480] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1176.507676] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1178.167268] Key type lgssc registered [ 1179.184151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.599530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.619920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.026569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.726301] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1197.540033] LustreError: 44987:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.583736] LustreError: 44987:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1201.606844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.995502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1206.387588] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1213.674420] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1220.697745] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1228.019548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1228.477749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.555572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1234.562804] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1242.268568] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1253.345229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.359531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.376672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.265804] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.12@tcp (stopping) [ 1257.457164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.457879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.474266] Lustre: Skipped 1 previous similar message [ 1257.483880] Lustre: Skipped 2 previous similar messages [ 1259.282087] Lustre: server umount lustre-MDT0000 complete [ 1260.323492] LustreError: 44983:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.562414] LustreError: 44986:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.588611] LustreError: 44986:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1263.480096] LustreError: 44967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138520 with bad export cookie 1443418588508322478 [ 1263.492395] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.496232] LustreError: 44967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1265.429964] LustreError: 44981:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.460571] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.12@tcp (stopping) [ 1267.681732] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.962776] Lustre: server umount lustre-MDT0001 complete [ 1285.897538] Lustre: server umount lustre-OST0000 complete [ 1305.733508] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1309.454599] Key type lgssc unregistered [ 1309.846851] LNet: 48850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.852256] LNetError: 48850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.861884] LNet: Removed LNI 192.168.203.112@tcp [ 1310.687732] Key type .llcrypt unregistered [ 1310.691697] Key type ._llcrypt unregistered [ 1335.084925] Key type ._llcrypt registered [ 1335.087658] Key type .llcrypt registered [ 1335.185969] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1336.686037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1336.733222] alg: No test for adler32 (adler32-zlib) [ 1337.781643] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1338.012690] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1339.679170] Key type lgssc registered [ 1340.903936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.331227] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.352476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.961988] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1370.079576] LustreError: 50235:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 1396.831430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.764989] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1413.473237] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1578.550580] Lustre: server umount lustre-OST0000 complete [ 1595.709577] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1598.532392] Key type lgssc unregistered [ 1598.829772] LNet: 51919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.836113] LNetError: 51919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.912479] LNet: Removed LNI 192.168.203.112@tcp [ 1600.638087] Key type .llcrypt unregistered [ 1600.643615] Key type ._llcrypt unregistered [ 1623.999928] Key type ._llcrypt registered [ 1624.004463] Key type .llcrypt registered [ 1624.119598] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1625.684986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.768219] alg: No test for adler32 (adler32-zlib) [ 1626.881662] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1627.316402] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1629.039267] Key type lgssc registered [ 1630.465706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1638.548739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1638.572335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.055610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.339496] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1649.638289] LustreError: 53335:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.672044] LustreError: 53335:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1654.439025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.869254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1660.066070] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1667.500284] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1673.298328] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1679.028428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1679.361451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.488412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1685.594343] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1692.021119] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1694.688950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1694.695777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.705096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1695.720505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1695.724018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.740638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.136372] Lustre: server umount lustre-OST0000 complete [ 1705.957036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.959077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.970310] Lustre: Skipped 1 previous similar message [ 1705.973513] Lustre: Skipped 1 previous similar message [ 1709.649963] Lustre: server umount lustre-MDT0000 complete [ 1711.075212] LustreError: 53335:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.094697] LustreError: 53335:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 1712.901337] LustreError: 53316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138969 with bad export cookie 2989628048683314232 [ 1712.907107] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1712.923962] LustreError: 53316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1713.347651] Lustre: server umount lustre-MDT0001 complete [ 1723.332760] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1725.703942] Key type lgssc unregistered [ 1726.038009] LNet: 56866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.050844] LNetError: 56866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.069963] LNet: Removed LNI 192.168.203.112@tcp [ 1726.786990] Key type .llcrypt unregistered [ 1726.790237] Key type ._llcrypt unregistered [ 1749.218759] Key type ._llcrypt registered [ 1749.222759] Key type .llcrypt registered [ 1749.336485] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1750.662592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1750.713122] alg: No test for adler32 (adler32-zlib) [ 1751.894553] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1752.204068] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1753.879161] Key type lgssc registered [ 1755.006943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.267502] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1763.298435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.785429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1783.904226] LustreError: 58256:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 1810.591516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.576986] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1823.765455] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1829.923270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.090630] LustreError: 57592:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b364a6be x1873951073503488/t0(0) o250->MGC192.168.203.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1830.476133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.539906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1834.376490] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1835.490244] LustreError: 58281:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.507722] LustreError: 58281:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1840.618129] LustreError: 58282:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.132668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.496045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1849.138250] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1856.677487] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1862.733141] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1865.184768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.194896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.212183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.210739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.217267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.220069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.242575] Lustre: Skipped 1 previous similar message [ 1870.834167] Lustre: server umount lustre-OST0000 complete [ 1895.923785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1895.943053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.971699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.730735] Lustre: server umount lustre-MDT0000 complete [ 1902.048687] LustreError: 59296:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.061561] LustreError: 59296:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 1902.721692] LustreError: 58263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139159 with bad export cookie 14921906321912787948 [ 1902.725512] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.730364] LustreError: 58263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1916.895266] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1917.286493] Lustre: server umount lustre-MDT0001 complete [ 1928.696770] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1931.364362] Key type lgssc unregistered [ 1931.620083] LNet: 62006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.629980] LNetError: 62006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.661765] LNet: Removed LNI 192.168.203.112@tcp [ 1932.646116] Key type .llcrypt unregistered [ 1932.648375] Key type ._llcrypt unregistered [ 1953.542444] Key type ._llcrypt registered [ 1953.546990] Key type .llcrypt registered [ 1953.689370] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1954.820450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.862855] alg: No test for adler32 (adler32-zlib) [ 1956.027201] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1956.269825] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1957.999182] Key type lgssc registered [ 1959.178389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1966.545538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1966.580904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.070941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.608615] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1978.337304] LustreError: 63400:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.352584] LustreError: 63400:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 1980.523403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.798156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1984.768067] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1992.001787] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1997.846807] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2004.993815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2005.390676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.564976] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2013.642969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2019.439291] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.216128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.225044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.240129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.057622] Lustre: server umount lustre-OST0000 complete [ 2044.385663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.397033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.403142] Lustre: Skipped 1 previous similar message [ 2049.507783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.517410] Lustre: Skipped 2 previous similar messages [ 2052.667088] Lustre: server umount lustre-MDT0000 complete [ 2054.625918] LustreError: 63400:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.642670] LustreError: 63400:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2056.145890] LustreError: 63380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139313 with bad export cookie 14672481429982029814 [ 2056.154105] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.160196] LustreError: 63380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2056.465637] Lustre: server umount lustre-MDT0001 complete [ 2068.551918] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2071.423223] Key type lgssc unregistered [ 2071.773505] LNet: 66956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.786129] LNetError: 66956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.804394] LNet: Removed LNI 192.168.203.112@tcp [ 2072.641344] Key type .llcrypt unregistered [ 2072.643712] Key type ._llcrypt unregistered [ 2099.797507] Key type ._llcrypt registered [ 2099.800371] Key type .llcrypt registered [ 2099.893601] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2101.092141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.126291] alg: No test for adler32 (adler32-zlib) [ 2102.223075] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2102.458282] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2104.137057] Key type lgssc registered [ 2105.175474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.020727] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:36:10 (1787139370) [ 2120.424930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2120.438623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.963606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.947375] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2131.429450] LustreError: 68661:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.459773] LustreError: 68661:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2136.066797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.489552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.481579] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2148.218346] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.906633] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2161.364109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.716231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.824696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2169.035608] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2177.720159] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2187.589809] Lustre: Failing over lustre-MDT0000 [ 2187.751418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.759094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.787582] LustreError: 68660:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.951294] Lustre: server umount lustre-MDT0000 complete [ 2190.680702] LustreError: 70959:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.329846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.351887] Lustre: Skipped 1 previous similar message [ 2195.739474] LustreError: 68657:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.772451] LustreError: 68657:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 2198.866956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.056435] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.229456] Lustre: *** cfs_fail_loc=135, val=0*** [ 2199.351465] LustreError: 71748:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2199.361683] LustreError: 71748:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.112@tcp: cfg command failed: rc = -2 [ 2199.373394] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2199.373394] [ 2199.383106] LustreError: MGC192.168.203.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2199.399652] LustreError: 71737:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2199.410634] LustreError: 71737:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2199.427182] LustreError: 71737:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2199.434324] LustreError: 71737:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2199.457792] Lustre: server umount lustre-MDT0000 complete [ 2199.462273] LustreError: 71737:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2200.858250] LustreError: 68656:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.918049] LustreError: 68656:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 2205.932762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.029371] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2206.310137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.372448] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2207.047916] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2211.317277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2211.351648] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2211.404269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2211.496265] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2221.565151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.568753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2221.579713] Lustre: Skipped 1 previous similar message [ 2226.179660] Lustre: server umount lustre-OST0000 complete [ 2231.780040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2231.783879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.789456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.789469] Lustre: Skipped 1 previous similar message [ 2231.810582] Lustre: Skipped 1 previous similar message [ 2236.261484] Lustre: server umount lustre-MDT0000 complete [ 2236.900948] LustreError: 68660:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.915427] LustreError: 68660:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 5 previous similar messages [ 2239.699994] LustreError: 68640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139496 with bad export cookie 1647520898948883247 [ 2239.703189] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.713608] LustreError: 68640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2240.052895] Lustre: server umount lustre-MDT0001 complete [ 2249.449767] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2251.776878] Key type lgssc unregistered [ 2252.027552] LNet: 73532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.045069] LNetError: 73532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.068713] LNet: Removed LNI 192.168.203.112@tcp [ 2252.852196] Key type .llcrypt unregistered [ 2252.854753] Key type ._llcrypt unregistered [ 2271.358708] Key type ._llcrypt registered [ 2271.360500] Key type .llcrypt registered [ 2271.448685] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2272.091708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.149074] alg: No test for adler32 (adler32-zlib) [ 2273.172974] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2273.396708] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2275.035658] Key type lgssc registered [ 2276.037704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.205583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.236701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.788624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.711746] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2294.244046] LustreError: 74925:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.280082] LustreError: 74925:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2296.859511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.284638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.962292] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2308.805494] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.952686] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2319.762622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2320.171525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.214126] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2329.409361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2333.844480] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2339.108606] Lustre: Failing over lustre-MDT0000 [ 2339.391756] Lustre: server umount lustre-MDT0000 complete [ 2339.811970] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.825198] LustreError: 74926:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.828305] Lustre: Skipped 1 previous similar message [ 2339.856668] LustreError: 74926:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2344.940845] LustreError: 74920:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.953327] LustreError: 74920:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 2347.860695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.997581] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2348.160606] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2348.251284] LustreError: 78010:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2348.259159] LustreError: 78010:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.112@tcp: cfg command failed: rc = -9 [ 2348.266964] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2348.266964] [ 2348.283178] LustreError: MGC192.168.203.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2348.303337] LustreError: 77999:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2348.310831] LustreError: 77999:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2348.321312] LustreError: 77999:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2348.330144] LustreError: 77999:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2348.350957] Lustre: server umount lustre-MDT0000 complete [ 2348.355630] LustreError: 77999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2350.049065] LustreError: 77414:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.073337] LustreError: 77414:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 2353.862700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.949973] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.122433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.172258] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2355.996187] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2358.062394] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2359.281608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2359.304617] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2359.333869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2364.390724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.391695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2364.402723] Lustre: Skipped 2 previous similar messages [ 2366.061976] Lustre: server umount lustre-OST0000 complete [ 2369.505939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.517240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.518490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2369.525371] Lustre: Skipped 1 previous similar message [ 2369.549086] Lustre: Skipped 2 previous similar messages [ 2374.628102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.376744] Lustre: server umount lustre-MDT0000 complete [ 2379.323704] LustreError: 74905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139636 with bad export cookie 12163074128817431761 [ 2379.325511] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.332262] LustreError: 74905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2379.682169] Lustre: server umount lustre-MDT0001 complete [ 2395.495386] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2398.467417] Key type lgssc unregistered [ 2398.792272] LNet: 79792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.813668] LNetError: 79792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.832096] LNet: Removed LNI 192.168.203.112@tcp [ 2399.608306] Key type .llcrypt unregistered [ 2399.613315] Key type ._llcrypt unregistered [ 2410.864602] Key type ._llcrypt registered [ 2410.866772] Key type .llcrypt registered [ 2410.948921] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2411.158578] Key type .llcrypt unregistered [ 2411.163707] Key type ._llcrypt unregistered [ 2430.073499] Key type ._llcrypt registered [ 2430.077754] Key type .llcrypt registered [ 2430.152781] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2430.878552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2431.001337] alg: No test for adler32 (adler32-zlib) [ 2432.005961] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2432.229559] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2433.879385] Key type lgssc registered [ 2434.821780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2440.692229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.705250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.091953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.638808] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2452.448198] LustreError: 81678:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.472769] LustreError: 81678:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2453.805833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.187177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2458.420176] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2464.135083] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.649554] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.946336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.224414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.854537] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2487.408379] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2487.536719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2498.020168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.029839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.035127] Lustre: Skipped 1 previous similar message [ 2502.530697] Lustre: server umount lustre-OST0000 complete [ 2508.256519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.283253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.288146] Lustre: Skipped 1 previous similar message [ 2508.293580] Lustre: Skipped 1 previous similar message [ 2512.581422] Lustre: server umount lustre-MDT0000 complete [ 2513.377763] LustreError: 82406:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.393837] LustreError: 82406:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2516.568860] LustreError: 81658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139773 with bad export cookie 4163808222279786368 [ 2516.579322] LustreError: 81658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2516.580774] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.975809] Lustre: server umount lustre-MDT0001 complete [ 2528.572716] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2531.075692] Key type lgssc unregistered [ 2531.365559] LNet: 85212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.373322] LNetError: 85212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.399912] LNet: Removed LNI 192.168.203.112@tcp [ 2533.084314] Key type .llcrypt unregistered [ 2533.087452] Key type ._llcrypt unregistered [ 2556.693786] Key type ._llcrypt registered [ 2556.699717] Key type .llcrypt registered [ 2556.805870] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2558.142461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.201955] alg: No test for adler32 (adler32-zlib) [ 2559.256706] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2559.514371] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2561.231498] Key type lgssc registered [ 2562.419545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2570.233883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2570.302980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.780874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.400141] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2581.987635] LustreError: 86626:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.008431] LustreError: 86626:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2584.447319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.745882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2589.110112] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2596.930738] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2602.619133] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2609.035673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.325586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.804177] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2616.530292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2622.728412] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2631.137638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.147084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.161576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.935075] Lustre: server umount lustre-OST0000 complete [ 2636.260205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2636.266983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.277558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.281391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.287215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.403529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.415670] Lustre: Skipped 2 previous similar messages [ 2645.520632] Lustre: server umount lustre-MDT0000 complete [ 2647.524984] LustreError: 86622:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.551286] LustreError: 86622:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2649.001407] LustreError: 86607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139905 with bad export cookie 784301384986627292 [ 2649.014445] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2649.372731] Lustre: server umount lustre-MDT0001 complete [ 2659.037438] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2661.297475] Key type lgssc unregistered [ 2661.592968] LNet: 90161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.602410] LNetError: 90161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2661.615873] LNet: Removed LNI 192.168.203.112@tcp [ 2662.373400] Key type .llcrypt unregistered [ 2662.382203] Key type ._llcrypt unregistered [ 2686.136147] Key type ._llcrypt registered [ 2686.137186] Key type .llcrypt registered [ 2686.315180] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2688.117636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.194609] alg: No test for adler32 (adler32-zlib) [ 2689.487742] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2689.876413] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2691.567537] Key type lgssc registered [ 2693.700197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2702.300189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2702.335522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.663184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.143891] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2713.060893] LustreError: 91573:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.093406] LustreError: 91573:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2716.333982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.689775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.892900] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2729.327874] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2734.659674] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2741.424176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.927910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.082185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2748.534532] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2756.231870] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2771.941321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.944527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.954591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.989726] Lustre: Skipped 1 previous similar message [ 2775.992782] Lustre: server umount lustre-OST0000 complete [ 2782.177335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.178728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.207063] Lustre: Skipped 1 previous similar message [ 2782.222355] Lustre: Skipped 2 previous similar messages [ 2785.880561] Lustre: server umount lustre-MDT0000 complete [ 2787.297396] LustreError: 91573:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.314103] LustreError: 91573:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 2789.192471] LustreError: 91555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140046 with bad export cookie 17478509853360292348 [ 2789.203416] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.208752] LustreError: 91555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2789.479888] Lustre: server umount lustre-MDT0001 complete [ 2799.551893] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2802.883271] Key type lgssc unregistered [ 2803.219640] LNet: 95133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.237779] LNetError: 95133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2803.266532] LNet: Removed LNI 192.168.203.112@tcp [ 2804.274212] Key type .llcrypt unregistered [ 2804.279216] Key type ._llcrypt unregistered [ 2829.715713] Key type ._llcrypt registered [ 2829.717315] Key type .llcrypt registered [ 2829.830247] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2830.728869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.809764] alg: No test for adler32 (adler32-zlib) [ 2831.825750] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2832.034825] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2833.695286] Key type lgssc registered [ 2834.714926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2842.493756] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2842.526401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.144505] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2863.263328] LustreError: 96520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 2889.887189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.394768] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2906.029505] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2912.386788] Lustre: server umount lustre-OST0000 complete [ 2922.421795] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:49:38 (1787140178) [ 2929.438917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.825078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.142860] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2935.274603] LustreError: 98161:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.296732] LustreError: 98161:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 2939.368457] LustreError: 98162:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.775844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.249897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.711543] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2954.115875] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2959.777823] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2966.686266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.175484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.935069] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2974.411776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2981.626284] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2994.144232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.151182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.167567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.173380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2995.177297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.181868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.201890] Lustre: Skipped 1 previous similar message [ 2999.889435] Lustre: server umount lustre-OST0000 complete [ 3003.872044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3003.882576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.901404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.535645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.544491] Lustre: Skipped 3 previous similar messages [ 3015.650062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.659189] Lustre: Skipped 1 previous similar message [ 3018.207246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3018.397662] Lustre: server umount lustre-MDT0000 complete [ 3020.768737] LustreError: 98157:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.783815] LustreError: 98157:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 3022.097340] LustreError: 98143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140279 with bad export cookie 6688586506762647958 [ 3022.110796] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.115369] LustreError: 98143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3022.647618] Lustre: server umount lustre-MDT0001 complete [ 3033.218548] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3035.915482] Key type lgssc unregistered [ 3036.259222] LNet: 101859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.266640] LNetError: 101859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.290858] LNet: Removed LNI 192.168.203.112@tcp [ 3037.263077] Key type .llcrypt unregistered [ 3037.268854] Key type ._llcrypt unregistered [ 3079.186298] Key type ._llcrypt registered [ 3079.193773] Key type .llcrypt registered [ 3079.372665] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3080.410783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3080.552820] alg: No test for adler32 (adler32-zlib) [ 3081.781632] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 3082.142023] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3083.800988] Key type lgssc registered [ 3084.834457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3092.713855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3092.748096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.317461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.250427] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3104.762145] LustreError: 104044:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.785406] LustreError: 104044:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 3108.145718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.614432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3112.845774] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3119.837491] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3125.829117] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3132.834455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.301957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.254845] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3139.499391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3146.344257] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3160.032156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3160.040539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.044792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.067456] Lustre: Skipped 1 previous similar message [ 3160.079266] Lustre: Skipped 1 previous similar message [ 3162.714421] Lustre: server umount lustre-OST0000 complete [ 3170.275304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.283268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3170.288775] Lustre: Skipped 1 previous similar message [ 3170.288792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.312669] LustreError: Skipped 1 previous similar message [ 3172.753348] Lustre: server umount lustre-MDT0000 complete [ 3175.395168] LustreError: 104043:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.402270] LustreError: 104043:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 3176.770194] LustreError: 104024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140433 with bad export cookie 4362622134795691212 [ 3176.777458] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.020624] Lustre: server umount lustre-MDT0001 complete [ 3187.917584] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3191.499526] Key type lgssc unregistered [ 3191.834869] LNet: 107577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3191.854938] LNetError: 107577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3191.885658] LNet: Removed LNI 192.168.203.112@tcp [ 3193.121841] Key type .llcrypt unregistered [ 3193.123844] Key type ._llcrypt unregistered [ 3213.612872] Key type ._llcrypt registered [ 3213.618428] Key type .llcrypt registered [ 3213.750991] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3215.137552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.153836] alg: No test for adler32 (adler32-zlib) [ 3216.197501] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 3216.406534] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3218.064181] Key type lgssc registered [ 3219.252652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.423427] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.446769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.975744] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3248.095343] LustreError: 108824:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 3274.783675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.010913] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3289.146409] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3297.457589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.637557] LustreError: MGC192.168.203.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3297.646129] LustreError: 109847:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3297.650461] LustreError: 109847:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3297.656093] LustreError: 109847:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3297.714489] Lustre: server umount lustre-MDT0000 complete [ 3297.716554] LustreError: 109847:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3316.817830] Lustre: server umount lustre-OST0000 complete [ 3326.850611] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 3337.887769] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3375.399350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.666867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3375.689576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3375.749537] Lustre: lustre-MDT0000: new disk, initializing [ 3375.837836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.858824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3380.448201] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3391.061491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.202823] Lustre: 112638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3391.260825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3391.265664] Lustre: Skipped 1 previous similar message [ 3391.370701] Lustre: lustre-MDT0001: new disk, initializing [ 3391.465750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.502411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3391.514812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3396.065878] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3406.422591] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3412.659403] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3420.432861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.690910] Lustre: lustre-OST0000: new disk, initializing [ 3420.694938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3420.700877] Lustre: 114608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3420.806551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.693497] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3428.451291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3428.457303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3428.592165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3438.097537] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3443.437279] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3443.622828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3449.169809] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3449.439696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3454.432966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.436623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.441485] LustreError: Skipped 1 previous similar message [ 3454.444514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.456066] Lustre: Skipped 1 previous similar message [ 3457.823685] Lustre: server umount lustre-OST0000 complete [ 3463.137438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3463.149732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.185995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.197851] Lustre: Skipped 1 previous similar message [ 3464.674338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.674444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.692472] Lustre: Skipped 1 previous similar message [ 3468.168967] Lustre: server umount lustre-MDT0000 complete [ 3472.001940] LustreError: 112632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140728 with bad export cookie 704261661668877282 [ 3472.004464] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.012025] LustreError: 112632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3472.495501] Lustre: server umount lustre-MDT0001 complete [ 3481.284883] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:58:57 (1787140737) [ 3484.395595] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3501.797694] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 3512.112400] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3547.878677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.146919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3548.173252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3548.250190] Lustre: lustre-MDT0000: new disk, initializing [ 3548.331501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.354495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3552.823904] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3563.976944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.178987] Lustre: 119509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3564.270229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3564.273556] Lustre: Skipped 1 previous similar message [ 3564.383585] Lustre: lustre-MDT0001: new disk, initializing [ 3564.465714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.507078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3564.520772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3569.195453] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3579.580967] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3586.223297] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3591.623039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.819291] Lustre: lustre-OST0000: new disk, initializing [ 3591.823976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.834994] Lustre: 121474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3591.915234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.336372] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3598.395291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3598.411341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3598.591369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3608.585527] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.440780] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3614.676715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3619.669810] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3619.898272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3624.417948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3624.421817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.422534] LustreError: Skipped 1 previous similar message [ 3624.441333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3624.455225] Lustre: Skipped 1 previous similar message [ 3629.543453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.553120] Lustre: Skipped 1 previous similar message [ 3636.191110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3636.316194] Lustre: server umount lustre-OST0000 complete [ 3641.316182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3641.330929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.354196] Lustre: Skipped 1 previous similar message [ 3641.369704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.382987] Lustre: Skipped 2 previous similar messages [ 3646.659560] Lustre: server umount lustre-MDT0000 complete [ 3650.018522] LustreError: 119516:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.039719] LustreError: 119516:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 4 previous similar messages [ 3650.623449] LustreError: 119503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140907 with bad export cookie 704261661668878164 [ 3650.629270] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3650.632369] LustreError: 119503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3650.959294] Lustre: server umount lustre-MDT0001 complete [ 3660.342495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.831051] LustreError: 123708:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.943532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.820888] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3666.407681] LustreError: 123709:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.501376] LustreError: 123708:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.256879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.798720] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3686.295691] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3692.716956] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3698.960431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3699.105379] LustreError: 125529:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.108936] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3710.888448] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3718.479586] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3724.771228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3724.785167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.812459] Lustre: Skipped 1 previous similar message [ 3724.823575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.835511] Lustre: Skipped 2 previous similar messages [ 3729.999474] Lustre: server umount lustre-OST0000 complete [ 3736.034496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3736.042667] LustreError: Skipped 1 previous similar message [ 3748.319243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3748.606907] Lustre: server umount lustre-MDT0000 complete [ 3750.375693] LustreError: 123704:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3750.405887] LustreError: 123704:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 3752.599018] LustreError: 123690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141009 with bad export cookie 704261661668879046 [ 3752.608447] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.608595] LustreError: 123690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3752.880565] Lustre: server umount lustre-MDT0001 complete [ 3764.072615] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3768.355365] Key type lgssc unregistered [ 3768.861148] LNet: 127341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3768.876336] LNetError: 127341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3769.958631] LNet: Removed LNI 192.168.203.112@tcp [ 3770.881766] Key type .llcrypt unregistered [ 3770.887847] Key type ._llcrypt unregistered [ 3791.742682] Key type ._llcrypt registered [ 3791.746756] Key type .llcrypt registered [ 3791.925725] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 3807.748061] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3809.190898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3809.281822] alg: No test for adler32 (adler32-zlib) [ 3810.672440] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 3811.044454] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3812.903162] Key type lgssc registered [ 3814.536313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3854.241966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3854.306437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.666127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3855.722374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3855.842819] Lustre: lustre-MDT0000: new disk, initializing [ 3855.928728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.946746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3860.533378] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3872.470924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.562692] Lustre: 130501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3872.611657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3872.615182] Lustre: Skipped 1 previous similar message [ 3872.675815] Lustre: lustre-MDT0001: new disk, initializing [ 3872.768238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3872.798192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3872.810477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3877.072801] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3886.019257] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3891.933564] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3899.582959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.922428] Lustre: lustre-OST0000: new disk, initializing [ 3899.932887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3899.954659] Lustre: 132467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3900.059711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.334943] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3907.191540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3907.203103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3907.272575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3915.852281] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3920.339466] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3920.496202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3925.670927] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3925.934324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3928.033687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3928.037838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.038902] LustreError: Skipped 1 previous similar message [ 3928.067914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3928.083409] Lustre: Skipped 1 previous similar message [ 3933.169987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.187408] Lustre: Skipped 1 previous similar message [ 3938.278913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.298164] Lustre: Skipped 1 previous similar message [ 3941.855218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3942.061466] Lustre: server umount lustre-OST0000 complete [ 3948.512914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.528784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.536874] Lustre: Skipped 2 previous similar messages [ 3954.345388] Lustre: server umount lustre-MDT0000 complete [ 3958.039397] LustreError: 130492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141215 with bad export cookie 1009227109577638893 [ 3958.050561] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3958.380662] Lustre: server umount lustre-MDT0001 complete [ 3966.469961] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:07:03 (1787141223) [ 3974.054438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.653717] LustreError: 134882:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.685456] LustreError: 134882:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 3974.763210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.670247] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3979.748580] LustreError: 134883:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.865246] LustreError: 134882:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.309337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.713696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3993.364607] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4001.388268] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4007.789169] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.920900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.924884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.941593] Lustre: Skipped 1 previous similar message [ 4013.959326] Lustre: Skipped 2 previous similar messages [ 4024.288739] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4024.292898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4024.305624] Lustre: Skipped 3 previous similar messages [ 4024.422922] Lustre: server umount lustre-MDT0000 complete [ 4027.981400] LustreError: 134863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141284 with bad export cookie 1009227109577639775 [ 4027.981793] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.993052] LustreError: 134863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4028.425230] Lustre: server umount lustre-MDT0001 complete [ 4036.463199] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:08:12 (1787141292) [ 4042.847492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4058.466659] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4063.583413] LustreError: 137298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 4089.311831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.296654] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4103.526350] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.431455] Lustre: server umount lustre-OST0000 complete [ 4113.826985] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:09:30 (1787141370) [ 4121.246633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.881812] LustreError: 138744:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4122.013904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4126.773183] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4127.200489] LustreError: 138745:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.907142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4139.650210] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4146.970682] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4152.969657] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4159.766970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4160.169448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.183360] Lustre: Skipped 1 previous similar message [ 4167.047170] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4175.253421] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.608185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4186.614684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.642088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.710995] Lustre: server umount lustre-MDT0000 complete [ 4192.225860] LustreError: 138745:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.255528] LustreError: 138745:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 4194.768168] LustreError: 138724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141451 with bad export cookie 1009227109577640440 [ 4194.770279] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.784172] LustreError: 138724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4195.121721] Lustre: server umount lustre-MDT0001 complete [ 4208.388542] Lustre: server umount lustre-OST0000 complete [ 4217.416450] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:11:13 (1787141473) [ 4223.994918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.394748] LustreError: 142399:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.482325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.249596] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4237.126447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.169847] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4249.825713] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4255.942388] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4262.997433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.565156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4269.978491] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4278.740913] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4283.935953] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4284.171285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4288.248861] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4288.470863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4293.094325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.098181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.104014] Lustre: Skipped 2 previous similar messages [ 4293.113250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.120709] Lustre: Skipped 4 previous similar messages [ 4296.673492] Lustre: server umount lustre-OST0000 complete [ 4303.328507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.341169] Lustre: Skipped 2 previous similar messages [ 4308.192869] Lustre: server umount lustre-MDT0000 complete [ 4308.452601] LustreError: 144756:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.488549] LustreError: 144756:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 4312.299722] LustreError: 142381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141569 with bad export cookie 1009227109577641511 [ 4312.308031] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4312.332133] LustreError: 142381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4312.606930] Lustre: server umount lustre-MDT0001 complete [ 4321.609229] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:12:58 (1787141578) [ 4327.649563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.265139] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4348.385101] LustreError: 146559:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 4374.111282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.124531] Lustre: Skipped 2 previous similar messages [ 4382.759988] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4391.159213] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4398.270910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.776805] LustreError: 146584:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.936259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4403.768388] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4412.330721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.007271] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4426.428074] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4432.969329] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4437.915445] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4438.162494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4442.550651] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4442.801391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4448.736904] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.743835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.744239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4448.761955] Lustre: Skipped 2 previous similar messages [ 4448.788358] Lustre: Skipped 4 previous similar messages [ 4451.193766] Lustre: server umount lustre-MDT0000 complete [ 4455.019166] LustreError: 149706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141711 with bad export cookie 1009227109577642533 [ 4455.020090] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.026460] LustreError: 149706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4455.444693] Lustre: server umount lustre-MDT0001 complete [ 4470.087970] Lustre: server umount lustre-OST0000 complete [ 4479.765738] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 08:15:36 (1787141736) [ 4488.329752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.034167] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4509.153664] LustreError: 150757:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 4540.632939] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4548.512196] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4555.325945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.683665] LustreError: 150782:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.701867] LustreError: 150782:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 6 previous similar messages [ 4555.828138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4560.876884] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4569.772154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.918142] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4583.693122] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4590.580461] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4597.867847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4598.030617] Lustre: lustre-OST0001: new disk, initializing [ 4598.033848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4598.039374] Lustre: 153629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4603.441800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4603.449879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4603.565919] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4604.406619] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4613.912594] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4617.936227] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4618.164199] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4622.685556] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4622.918286] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4626.403918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4626.412165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.429933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.199361] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4639.267334] Lustre: server umount lustre-OST0000 complete [ 4644.846802] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4644.851614] LustreError: Skipped 2 previous similar messages [ 4649.777114] Lustre: server umount lustre-OST0001 complete [ 4659.982424] Lustre: server umount lustre-MDT0000 complete [ 4664.640584] LustreError: 150765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141921 with bad export cookie 1009227109577643485 [ 4664.649670] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4664.659789] LustreError: 150765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4665.305962] Lustre: server umount lustre-MDT0001 complete [ 4689.770152] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 08:19:06 (1787141946) [ 4691.691389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4693.505196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4695.345510] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 08:19:11 (1787141951) [ 4702.771834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.114456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4703.155995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4703.361452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.367519] Lustre: Skipped 6 previous similar messages [ 4708.385180] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4718.019154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4718.132491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4718.176402] Lustre: 157190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4723.140750] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4731.643474] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4737.499218] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4745.002225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4745.270743] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4752.175377] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4754.063260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4760.928759] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4766.171508] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4766.402714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4771.444544] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4771.649749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4774.371564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4774.380815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.401637] Lustre: Skipped 5 previous similar messages [ 4788.191134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4788.425241] Lustre: server umount lustre-OST0000 complete [ 4811.999687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4818.562469] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.588914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.589552] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4818.613905] Lustre: Skipped 1 previous similar message [ 4819.454191] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4828.040081] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4833.832759] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4834.044306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4838.189724] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4838.348417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4844.767937] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b5bd0331000.ost_server_uuid 50 [ 4846.379435] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b5bd0331000.ost_server_uuid in FULL state after 0 sec [ 4846.608297] LustreError: 157199:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.12@tcp) returned error from blocking AST (req@0000000016c62b44 x1873953232789120 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a5c5e475/0xe017eb58411df37 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.12@tcp remote: 0xc6c17c37d8ee74d0 expref: 5 pid: 157199 timeout: 4946 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4846.653415] LustreError: lustre-MDT0000: A client on nid 192.168.203.12@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4846.666895] LustreError: 157188:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.12@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a5c5e475/0xe017eb58411df37 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.12@tcp remote: 0xc6c17c37d8ee74d0 expref: 6 pid: 157199 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4852.711620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4857.643913] Lustre: server umount lustre-OST0000 complete [ 4876.255137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4876.598989] Lustre: server umount lustre-MDT0000 complete [ 4879.334312] LustreError: 157198:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.355860] LustreError: 157198:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 13 previous similar messages [ 4880.703777] LustreError: 159053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142137 with bad export cookie 1009227109577644668 [ 4880.705510] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.713226] LustreError: 159053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4895.199441] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4895.436751] Lustre: server umount lustre-MDT0001 complete [ 4906.167512] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 4908.853683] Key type lgssc unregistered [ 4909.196948] LNet: 163323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4909.213973] LNetError: 163323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4909.240585] LNet: Removed LNI 192.168.203.112@tcp [ 4910.218481] Key type .llcrypt unregistered [ 4910.221450] Key type ._llcrypt unregistered [ 4936.216771] Key type ._llcrypt registered [ 4936.225750] Key type .llcrypt registered [ 4936.360152] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 4937.805233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.890926] alg: No test for adler32 (adler32-zlib) [ 4939.082698] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 4939.324592] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 4941.023190] Key type lgssc registered [ 4942.206793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4950.537084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4950.588727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.004568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.460192] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4961.258592] LustreError: 164745:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4961.271677] LustreError: 164745:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 4966.370768] LustreError: 164744:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.526102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.186335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4973.442505] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4982.064496] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4988.526187] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4995.206392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4995.607283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.682126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5001.987820] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5010.364461] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5017.810154] Lustre: Failing over lustre-MDT0000 [ 5018.084105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.088023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.112505] Lustre: Skipped 1 previous similar message [ 5018.143142] Lustre: Skipped 2 previous similar messages [ 5018.295839] Lustre: server umount lustre-MDT0000 complete [ 5018.953282] LustreError: 164739:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.203800] LustreError: 164740:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.224838] LustreError: 164740:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5028.327741] LustreError: 166651:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.344638] LustreError: 166651:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5031.334904] LustreError: 164724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.12@tcp arrived at 1787142288 with bad export cookie 12985139972616821774 [ 5031.358946] LustreError: 164724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5037.801909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.897475] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.156501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.215111] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5042.235176] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5043.177509] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5043.189593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5044.462900] Lustre: lustre-MDT0000: Denying connection for new client 50948f6f-0086-4c2b-b660-e4b7dcae5f4d (at 192.168.203.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5053.408350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.414816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5053.417530] Lustre: Skipped 2 previous similar messages [ 5053.426862] Lustre: Skipped 1 previous similar message [ 5058.518643] Lustre: server umount lustre-OST0000 complete [ 5063.048234] LustreError: 168474:0:(ldlm_lib.c:3008:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5063.063048] Lustre: 167824:0:(ldlm_lib.c:2408:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5063.079742] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5063.092858] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5063.100223] LustreError: 167824:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000989790fb x1873954415358208/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5063.111213] LustreError: 167824:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5063.117792] LustreError: 167824:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5063.124498] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5063.427917] Lustre: server umount lustre-MDT0000 complete [ 5063.651983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.668823] LustreError: 164739:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.679865] LustreError: 164739:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 5067.516925] LustreError: 164726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142324 with bad export cookie 12985139972616821879 [ 5067.526064] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.534404] LustreError: 164726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5067.599241] LustreError: 164053:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5070.095683] Lustre: server umount lustre-MDT0001 complete [ 5081.227470] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5084.221554] Key type lgssc unregistered [ 5084.482523] LNet: 169230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5084.494067] LNetError: 169230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5084.509212] LNet: Removed LNI 192.168.203.112@tcp [ 5085.356653] Key type .llcrypt unregistered [ 5085.359446] Key type ._llcrypt unregistered [ 5108.063399] Key type ._llcrypt registered [ 5108.065561] Key type .llcrypt registered [ 5108.197772] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5109.206777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5109.261875] alg: No test for adler32 (adler32-zlib) [ 5110.271267] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 5110.543582] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5112.192082] Key type lgssc registered [ 5113.254539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5121.366491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5121.385693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.737791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.109934] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5132.260722] LustreError: 170642:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.281017] LustreError: 170642:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5135.059113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.455659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5139.667945] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5147.260568] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5153.440056] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5160.031247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.442516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.689969] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5167.603663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5173.000508] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5181.920609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5181.928580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.940649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.946813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.950270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5182.963983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.152631] Lustre: server umount lustre-OST0000 complete [ 5191.648208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5191.653246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.660445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.664996] Lustre: Skipped 1 previous similar message [ 5195.953618] Lustre: server umount lustre-MDT0000 complete [ 5198.310211] LustreError: 170643:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.344208] LustreError: 170643:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 5200.594761] LustreError: 170623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142457 with bad export cookie 9678211420408293861 [ 5200.608782] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5200.618831] LustreError: 170623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5201.147574] Lustre: server umount lustre-MDT0001 complete [ 5214.113399] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5216.764565] Key type lgssc unregistered [ 5217.222985] LNet: 174178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5217.234446] LNetError: 174178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5217.252865] LNet: Removed LNI 192.168.203.112@tcp [ 5218.017700] Key type .llcrypt unregistered [ 5218.023725] Key type ._llcrypt unregistered [ 5253.828133] Key type ._llcrypt registered [ 5253.830132] Key type .llcrypt registered [ 5253.935232] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5254.917806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5254.955128] alg: No test for adler32 (adler32-zlib) [ 5256.048511] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 5256.279514] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5257.959791] Key type lgssc registered [ 5259.015353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5266.690715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5266.702711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5268.178809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.741358] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5278.721606] LustreError: 175881:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.755022] LustreError: 175881:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5281.682428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5282.072683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5286.575945] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5294.330980] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5299.995906] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5306.211588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.480027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.385660] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5313.649100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5319.196680] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5327.435824] Lustre: server umount lustre-OST0000 complete [ 5333.478916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5333.488428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.504130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.497859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.499784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.624192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5339.634887] Lustre: Skipped 2 previous similar messages [ 5341.779261] Lustre: server umount lustre-MDT0000 complete [ 5344.736728] LustreError: 175882:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.750742] Lustre: 175192:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787142585/real 1787142585] req@00000000964dfdda x1873954747738112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787142601 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5344.797263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.574489] LustreError: 175862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142602 with bad export cookie 3711293078659699977 [ 5345.578710] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5345.588607] LustreError: 175862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5345.960445] Lustre: server umount lustre-MDT0001 complete [ 5357.274336] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5361.051034] Key type lgssc unregistered [ 5361.438290] LNet: 179416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5361.451224] LNetError: 179416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5361.470912] LNet: Removed LNI 192.168.203.112@tcp [ 5362.427527] Key type .llcrypt unregistered [ 5362.435090] Key type ._llcrypt unregistered [ 5386.404751] Key type ._llcrypt registered [ 5386.408226] Key type .llcrypt registered [ 5386.547819] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5387.624570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5387.766155] alg: No test for adler32 (adler32-zlib) [ 5388.886597] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 5389.212925] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5390.930065] Key type lgssc registered [ 5392.399921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5403.031827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.090888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.563282] Lustre: *** cfs_fail_loc=135, val=0*** [ 5404.702605] LustreError: 180945:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5404.717832] LustreError: 180945:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.112@tcp: cfg command failed: rc = -2 [ 5404.727660] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5404.727660] [ 5404.740761] LustreError: MGC192.168.203.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5404.766520] LustreError: 180906:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5404.771163] LustreError: 180906:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5404.776479] LustreError: 180906:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5404.782258] LustreError: 180906:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5404.852637] Lustre: server umount lustre-MDT0000 complete [ 5404.856591] LustreError: 180906:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5412.505132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.988299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.538400] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5422.563672] LustreError: 181315:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.590866] LustreError: 181315:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5427.283859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.679198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5432.222536] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5440.796904] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5447.580699] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5452.710676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.725919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.731784] Lustre: Skipped 1 previous similar message [ 5452.750665] Lustre: Skipped 1 previous similar message [ 5456.000244] Lustre: server umount lustre-MDT0000 complete [ 5457.890156] LustreError: 182043:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.911900] LustreError: 182043:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 5459.356145] LustreError: 181297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142716 with bad export cookie 4748484980364322956 [ 5459.372238] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.672881] Lustre: server umount lustre-MDT0001 complete [ 5469.870217] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5472.431838] Key type lgssc unregistered [ 5472.707681] LNet: 183641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5472.729716] LNetError: 183641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5472.754133] LNet: Removed LNI 192.168.203.112@tcp [ 5473.488950] Key type .llcrypt unregistered [ 5473.492104] Key type ._llcrypt unregistered [ 5496.272828] Key type ._llcrypt registered [ 5496.275573] Key type .llcrypt registered [ 5496.387586] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5496.665564] Key type .llcrypt unregistered [ 5496.670043] Key type ._llcrypt unregistered [ 5514.300894] Key type ._llcrypt registered [ 5514.304676] Key type .llcrypt registered [ 5514.453804] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5515.379767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5515.472416] alg: No test for adler32 (adler32-zlib) [ 5516.670480] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 5516.975568] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5518.688065] Key type lgssc registered [ 5519.913675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5528.870148] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5528.889307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5544.543477] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5549.664771] LustreError: 185783:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 5576.354645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.842187] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5589.244314] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5595.620181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5596.070992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.135039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5600.690842] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5601.250127] LustreError: 185809:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.265499] LustreError: 185809:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 5606.377084] LustreError: 185810:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.161980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.618621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.767450] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5619.964243] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5626.112985] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5628.834942] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5642.210591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.221556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.239337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.722933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.749967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.842058] Lustre: server umount lustre-OST0000 complete [ 5652.963222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.984801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.001799] Lustre: Skipped 1 previous similar message [ 5656.712059] Lustre: server umount lustre-MDT0000 complete [ 5658.080290] LustreError: 186826:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.097416] LustreError: 186826:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 5660.423212] LustreError: 185791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142917 with bad export cookie 5351593513376111847 [ 5660.425987] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.447930] LustreError: 185791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5660.743631] Lustre: server umount lustre-MDT0001 complete [ 5672.347790] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5675.816411] Key type lgssc unregistered [ 5676.265905] LNet: 189796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5676.283397] LNetError: 189796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5676.309541] LNet: Removed LNI 192.168.203.112@tcp [ 5677.528732] Key type .llcrypt unregistered [ 5677.538122] Key type ._llcrypt unregistered [ 5704.919039] Key type ._llcrypt registered [ 5704.924405] Key type .llcrypt registered [ 5705.066189] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5706.120099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5706.182070] alg: No test for adler32 (adler32-zlib) [ 5707.361738] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 5707.657561] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5709.383234] Key type lgssc registered [ 5710.523580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5718.468898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5718.486738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.976642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5723.472237] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5729.252606] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.275116] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5731.980880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5732.330958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.562778] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5743.819142] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5749.680913] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5756.927497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5757.340318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.736840] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5764.531324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5773.208393] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5778.865485] Lustre: Failing over lustre-MDT0000 [ 5779.248891] Lustre: server umount lustre-MDT0000 complete [ 5779.941722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.951858] LustreError: 191941:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.986467] Lustre: Skipped 1 previous similar message [ 5780.052944] LustreError: 191941:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5781.318827] LustreError: 191210:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.062867] LustreError: 191208:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.096262] LustreError: 191208:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5790.177566] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.193537] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 3 previous similar messages [ 5795.300129] Lustre: 190526:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143036/real 1787143036] req@00000000a585abe4 x1873955220646400/t0(0) o400->MGC192.168.203.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787143052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5795.337445] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5798.592199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.416878] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.447624] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 8 previous similar messages [ 5805.542164] LustreError: 190522:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ba302522 x1873955220651264/t0(0) o250->MGC192.168.203.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5805.916715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.993209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5807.900906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5810.267207] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5811.201996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5811.235398] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5811.317271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5814.391137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5821.554798] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5835.238400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5835.249041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.264320] Lustre: Skipped 1 previous similar message [ 5835.283389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.771785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.775492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.063182] Lustre: server umount lustre-OST0000 complete [ 5847.010786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.013115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5847.038594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.045320] Lustre: Skipped 2 previous similar messages [ 5851.108829] Lustre: server umount lustre-MDT0000 complete [ 5852.129228] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.150780] LustreError: 191213:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 6 previous similar messages [ 5854.517735] LustreError: 193058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143111 with bad export cookie 7524123261687660785 [ 5854.529792] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5854.538075] LustreError: 193058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5854.787822] Lustre: server umount lustre-MDT0001 complete [ 5865.227158] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5868.292385] Key type lgssc unregistered [ 5868.561994] LNet: 196122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5868.568206] LNetError: 196122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5868.580165] LNet: Removed LNI 192.168.203.112@tcp [ 5869.284558] Key type .llcrypt unregistered [ 5869.287290] Key type ._llcrypt unregistered [ 5894.540686] Key type ._llcrypt registered [ 5894.543346] Key type .llcrypt registered [ 5894.697238] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5895.476653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5895.507164] alg: No test for adler32 (adler32-zlib) [ 5896.574449] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 5896.835482] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5898.487132] Key type lgssc registered [ 5899.596480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5906.135988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5906.147345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.693894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.710659] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5917.158338] LustreError: 197537:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.189757] LustreError: 197537:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 5920.063994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.531638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5925.503478] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5932.495561] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5937.883956] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5944.514901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.878229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.503763] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5953.087547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5957.744973] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5961.426937] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5966.697240] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5983.425578] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5991.912768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5991.923302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.948436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5993.954848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5993.973875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5993.984603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.639655] Lustre: server umount lustre-OST0000 complete [ 6002.665889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6002.695049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.721637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.728422] Lustre: Skipped 1 previous similar message [ 6006.563311] Lustre: server umount lustre-MDT0000 complete [ 6009.326853] LustreError: 198265:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.348827] LustreError: 198265:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 6010.230028] LustreError: 197519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143267 with bad export cookie 1774829771219245921 [ 6010.238717] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.244415] LustreError: 197519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6010.600327] Lustre: server umount lustre-MDT0001 complete [ 6021.136035] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 6024.046751] Key type lgssc unregistered [ 6024.471734] LNet: 201405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.485444] LNetError: 201405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.500806] LNet: Removed LNI 192.168.203.112@tcp [ 6025.573885] Key type .llcrypt unregistered [ 6025.578377] Key type ._llcrypt unregistered [ 6044.406732] Key type ._llcrypt registered [ 6044.408768] Key type .llcrypt registered [ 6044.542934] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6045.669481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6045.718114] alg: No test for adler32 (adler32-zlib) [ 6046.800669] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 6047.022440] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 6048.704062] Key type lgssc registered [ 6049.715542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6056.578900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6056.588413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.970225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.092278] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6067.172379] LustreError: 202669:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.204067] LustreError: 202669:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 6070.963229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.263821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.036883] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6083.564713] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6089.621568] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.778695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6097.169442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.320651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6103.571563] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6112.076887] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.211146] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6131.200509] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6140.873266] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6151.298179] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6162.918468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.927020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.938359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.978562] Lustre: Skipped 1 previous similar message [ 6168.034195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.050723] Lustre: Skipped 2 previous similar messages [ 6173.151976] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6173.158162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.179947] Lustre: Skipped 1 previous similar message [ 6173.424758] Lustre: server umount lustre-OST0000 complete [ 6178.275932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.281806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.304000] Lustre: Skipped 1 previous similar message [ 6178.331222] Lustre: Skipped 1 previous similar message [ 6183.393053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6183.411144] Lustre: Skipped 1 previous similar message [ 6184.106913] Lustre: server umount lustre-MDT0000 complete [ 6187.916439] LustreError: 205349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143444 with bad export cookie 1679051705362654795 [ 6187.922718] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.928454] LustreError: 205349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6188.247299] Lustre: server umount lustre-MDT0001 complete [ 6198.438755] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 6200.862224] Key type lgssc unregistered [ 6201.137762] LNet: 207746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6201.146041] LNetError: 207746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6201.161354] LNet: Removed LNI 192.168.203.112@tcp [ 6202.104545] Key type .llcrypt unregistered [ 6202.105940] Key type ._llcrypt unregistered [ 6226.735324] Key type ._llcrypt registered [ 6226.741771] Key type .llcrypt registered [ 6226.840581] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6228.147685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6228.170356] alg: No test for adler32 (adler32-zlib) [ 6229.300194] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 6229.578684] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 6231.231166] Key type lgssc registered [ 6232.315217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6240.882497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6240.907806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.316111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.798069] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6252.517752] LustreError: 209158:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.537936] LustreError: 209158:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 6255.862050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6256.294910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.056503] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6269.055605] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6274.734783] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6281.918056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.418208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.556689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6288.536284] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6296.585264] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6313.440150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.455648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.474742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6317.539675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.543662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6317.577217] Lustre: Skipped 1 previous similar message [ 6318.956712] Lustre: server umount lustre-OST0000 complete [ 6322.656677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.677607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6322.685732] Lustre: Skipped 1 previous similar message [ 6327.781711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6327.798949] Lustre: Skipped 1 previous similar message [ 6328.833464] Lustre: server umount lustre-MDT0000 complete [ 6332.896992] LustreError: 209159:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.923859] LustreError: 209159:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 6332.986132] LustreError: 209140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143589 with bad export cookie 8987712313384018796 [ 6333.004785] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.021149] LustreError: 209140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6333.425505] Lustre: server umount lustre-MDT0001 complete [ 6345.432602] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 6348.328289] Key type lgssc unregistered [ 6348.580784] LNet: 212843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6348.589285] LNetError: 212843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6348.617988] LNet: Removed LNI 192.168.203.112@tcp [ 6349.415248] Key type .llcrypt unregistered [ 6349.417525] Key type ._llcrypt unregistered [ 6368.720982] Key type ._llcrypt registered [ 6368.724815] Key type .llcrypt registered [ 6368.860275] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6369.759987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6369.881815] alg: No test for adler32 (adler32-zlib) [ 6370.992509] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 6371.262111] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 6372.920096] Key type lgssc registered [ 6374.062199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6381.529579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6381.553756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.977983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.906176] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6392.291829] LustreError: 214112:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.312146] LustreError: 214112:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 6396.466985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.908976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6401.634612] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6409.121084] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6415.047532] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6423.069443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6423.601201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6427.703288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6430.855827] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6441.438027] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6462.436900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6462.447729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.464339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6462.476143] Lustre: Skipped 1 previous similar message [ 6465.554598] Lustre: server umount lustre-OST0000 complete [ 6472.675859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.687249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.695606] Lustre: Skipped 2 previous similar messages [ 6475.781909] Lustre: server umount lustre-MDT0000 complete [ 6477.792261] LustreError: 214112:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.808889] LustreError: 214112:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 6479.403156] LustreError: 214093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143736 with bad export cookie 11048788124845276727 [ 6479.414135] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.415177] LustreError: 214093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6479.946960] Lustre: server umount lustre-MDT0001 complete [ 6492.116950] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 6495.044384] Key type lgssc unregistered [ 6495.355968] LNet: 217797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6495.368342] LNetError: 217797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6495.395043] LNet: Removed LNI 192.168.203.112@tcp [ 6496.589389] Key type .llcrypt unregistered [ 6496.591652] Key type ._llcrypt unregistered [ 6520.992083] Key type ._llcrypt registered [ 6520.997615] Key type .llcrypt registered [ 6521.138961] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6522.664205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6522.731095] alg: No test for adler32 (adler32-zlib) [ 6523.999513] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 6524.378901] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 6526.056723] Key type lgssc registered [ 6527.683975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6536.229349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6536.240541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.785407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6542.904579] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6547.947555] LustreError: 219066:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.980650] LustreError: 219066:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 6552.010325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.490246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6556.820421] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6564.603931] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6570.347731] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6576.991445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.128267] LustreError: 220885:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.462827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.634912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6583.790885] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6591.668348] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.237889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6601.307495] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6601.526097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6608.049912] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6616.276611] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6617.909542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6629.878047] Lustre: Permanently deactivating lustre-OST0001 [ 6629.935460] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6637.111443] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6644.075441] Lustre: Permanently reactivating lustre-OST0001 [ 6644.088188] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6644.092348] Lustre: Skipped 2 previous similar messages [ 6652.490333] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.520387] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6652.534866] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6652.552777] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6652.617893] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6658.016653] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6658.026720] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6658.028749] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6658.040431] Lustre: Skipped 2 previous similar messages [ 6658.068364] Lustre: Skipped 1 previous similar message [ 6659.042870] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6659.057738] Lustre: Skipped 1 previous similar message [ 6661.737469] Lustre: server umount lustre-OST0001 complete [ 6664.167303] LustreError: 220888:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6664.201121] LustreError: 220888:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 6669.280373] LustreError: 221189:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6669.301854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.303574] LustreError: 221189:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 1 previous similar message [ 6669.310780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.321830] Lustre: Skipped 1 previous similar message [ 6669.350015] Lustre: Skipped 1 previous similar message [ 6671.971542] Lustre: server umount lustre-OST0000 complete [ 6679.522388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.545301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.552591] Lustre: Skipped 1 previous similar message [ 6684.640976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6684.652391] Lustre: Skipped 1 previous similar message [ 6690.271246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6690.522802] Lustre: server umount lustre-MDT0000 complete [ 6694.226032] LustreError: 222695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143951 with bad export cookie 4370370658612534439 [ 6694.226820] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.238219] LustreError: 222695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6694.799918] Lustre: server umount lustre-MDT0001 complete [ 6706.895461] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 6710.346578] Key type lgssc unregistered [ 6710.821476] LNet: 224126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6710.833628] LNetError: 224126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6710.857034] LNet: Removed LNI 192.168.203.112@tcp [ 6712.049530] Key type .llcrypt unregistered [ 6712.052888] Key type ._llcrypt unregistered [ 6754.867718] Key type ._llcrypt registered [ 6754.871195] Key type .llcrypt registered [ 6755.016572] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6756.402501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6756.418859] alg: No test for adler32 (adler32-zlib) [ 6757.599868] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 6757.881730] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 6759.599164] Key type lgssc registered [ 6760.674696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6767.983716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6768.004992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.323969] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6769.339975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6769.544901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.277587] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6783.890956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.961072] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6784.001313] Lustre: 226328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6784.265708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.743828] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6797.790559] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6803.372625] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6809.258504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6809.434382] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6809.640748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6815.859831] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6816.966333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6824.215555] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6828.585262] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6831.871578] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6841.935689] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6849.166843] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6859.198153] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6878.569335] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6878.574756] Lustre: Skipped 2 previous similar messages [ 6911.701914] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6911.709377] Lustre: Skipped 3 previous similar messages [ 6980.400176] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6980.412567] Lustre: Skipped 8 previous similar messages [ 7001.568083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7001.580042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.582858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7001.603357] Lustre: Skipped 1 previous similar message [ 7001.622410] Lustre: Skipped 1 previous similar message [ 7005.503325] Lustre: server umount lustre-OST0000 complete [ 7009.762087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7009.768269] LustreError: Skipped 1 previous similar message [ 7009.776879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.787128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.810687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.817841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.424501] Lustre: server umount lustre-MDT0000 complete [ 7019.175640] LustreError: 226322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144276 with bad export cookie 10036811934178865747 [ 7019.179716] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7019.198085] LustreError: 226322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7019.618874] Lustre: server umount lustre-MDT0001 complete [ 7032.357896] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7036.379436] Key type lgssc unregistered [ 7036.922773] LNet: 231143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7036.976918] LNetError: 231143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.023028] LNet: Removed LNI 192.168.203.112@tcp [ 7038.521402] Key type .llcrypt unregistered [ 7038.523841] Key type ._llcrypt unregistered [ 7068.421807] Key type ._llcrypt registered [ 7068.426856] Key type .llcrypt registered [ 7068.606771] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7070.084812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7070.122523] alg: No test for adler32 (adler32-zlib) [ 7071.522882] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 7071.755349] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7073.423157] Key type lgssc registered [ 7074.666993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7083.309195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7083.325935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.867511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.559115] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7094.245865] LustreError: 232556:0:(ldlm_lib.c:1195:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.272820] LustreError: 232556:0:(ldlm_lib.c:1195:target_handle_connect()) Skipped 2 previous similar messages [ 7098.685879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.072166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.460844] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7110.289231] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7115.795829] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7122.801585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7123.168207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7129.329315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7129.992892] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7137.525386] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7155.183340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7155.187054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.206186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7155.232049] Lustre: Skipped 1 previous similar message [ 7156.079742] Lustre: server umount lustre-OST0000 complete [ 7160.293031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7160.294743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.297671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7160.297678] Lustre: Skipped 1 previous similar message [ 7160.344947] Lustre: Skipped 1 previous similar message [ 7165.408933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7165.417035] Lustre: Skipped 2 previous similar messages [ 7165.972954] Lustre: server umount lustre-MDT0000 complete [ 7169.607931] LustreError: 232536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144426 with bad export cookie 17458981306639244816 [ 7169.609751] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7169.623596] LustreError: 232536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7170.149117] Lustre: server umount lustre-MDT0001 complete [ 7180.822131] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7183.377578] Key type lgssc unregistered [ 7183.655942] LNet: 236250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7183.667103] LNetError: 236250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7183.687278] LNet: Removed LNI 192.168.203.112@tcp [ 7184.562992] Key type .llcrypt unregistered [ 7184.565748] Key type ._llcrypt unregistered [ 7207.087336] Key type ._llcrypt registered [ 7207.092412] Key type .llcrypt registered [ 7207.216388] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7208.420052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7208.468081] alg: No test for adler32 (adler32-zlib) [ 7209.542264] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 7209.802063] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7211.535255] Key type lgssc registered [ 7213.120748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7485.512642] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7488.076800] Key type lgssc unregistered [ 7488.319889] LNet: 238292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.329403] LNetError: 238292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.362181] LNet: Removed LNI 192.168.203.112@tcp [ 7489.173372] Key type .llcrypt unregistered [ 7489.176101] Key type ._llcrypt unregistered [ 7516.109163] Key type ._llcrypt registered [ 7516.110942] Key type .llcrypt registered [ 7516.198931] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 7533.174511] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7534.497089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7534.545365] alg: No test for adler32 (adler32-zlib) [ 7535.790574] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 7536.064494] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7537.751210] Key type lgssc registered [ 7538.573457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7569.200937] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7330 sec ========== 09:07:05 (1787144825) [ 7571.310929] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:07:07 (1787144827) === [ 7575.339662] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:07:11 (1787144831) === [ 7614.389203] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7616.954150] Key type lgssc unregistered [ 7617.176527] LNet: 243531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7617.188610] LNetError: 243531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7617.206979] LNet: Removed LNI 192.168.203.112@tcp [ 7618.000397] Key type .llcrypt unregistered [ 7618.004040] Key type ._llcrypt unregistered