[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474829633 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003161] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021214] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025764] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028007] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.032303] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035285] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.041749] debug: unmapping init [mem 0xffffffff85859000-0xffffffff85860fff] [ 0.043174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052313] rcu: Hierarchical SRCU implementation. [ 0.054639] smp: Bringing up secondary CPUs ... [ 0.055592] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059353] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146028] node 0 deferred pages initialised in 81ms [ 0.150018] devtmpfs: initialized [ 0.151329] x86/mm: Memory block size: 128MB [ 0.156357] gcov: version magic: 0x41383552 [ 0.160201] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161148] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162263] pinctrl core: initialized pinctrl subsystem [ 0.163199] [ 0.163729] ************************************************************* [ 0.164024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165016] ** ** [ 0.166020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167014] ** ** [ 0.168013] ** This means that this kernel is built to expose internal ** [ 0.169014] ** IOMMU data structures, which may compromise security on ** [ 0.170015] ** your system. ** [ 0.171014] ** ** [ 0.172017] ** If you see this message and you are not debugging the ** [ 0.173017] ** kernel, report this immediately to your vendor! ** [ 0.174014] ** ** [ 0.175015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176014] ************************************************************* [ 0.177840] NET: Registered protocol family 16 [ 0.179714] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.187089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193020] cpuidle: using governor menu [ 0.194802] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198864] PCI: Using configuration type 1 for base access [ 0.201152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215093] cryptd: max_cpu_qlen set to 1000 [ 0.218287] ACPI: Added _OSI(Module Device) [ 0.220027] ACPI: Added _OSI(Processor Device) [ 0.221015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228949] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236241] ACPI: Interpreter enabled [ 0.238089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239018] ACPI: Using IOAPIC for interrupt routing [ 0.241142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245477] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257678] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269489] acpiphp: Slot [2] registered [ 0.271149] acpiphp: Slot [5] registered [ 0.273107] acpiphp: Slot [6] registered [ 0.274143] acpiphp: Slot [7] registered [ 0.276138] acpiphp: Slot [8] registered [ 0.278181] acpiphp: Slot [9] registered [ 0.280159] acpiphp: Slot [10] registered [ 0.282318] acpiphp: Slot [3] registered [ 0.284161] acpiphp: Slot [4] registered [ 0.285433] acpiphp: Slot [11] registered [ 0.287125] acpiphp: Slot [12] registered [ 0.288123] acpiphp: Slot [13] registered [ 0.289164] acpiphp: Slot [14] registered [ 0.291134] acpiphp: Slot [15] registered [ 0.292121] acpiphp: Slot [16] registered [ 0.294182] acpiphp: Slot [17] registered [ 0.295000] acpiphp: Slot [18] registered [ 0.295000] acpiphp: Slot [19] registered [ 0.297132] acpiphp: Slot [20] registered [ 0.298134] acpiphp: Slot [21] registered [ 0.299187] acpiphp: Slot [22] registered [ 0.301269] acpiphp: Slot [23] registered [ 0.303129] acpiphp: Slot [24] registered [ 0.304100] acpiphp: Slot [25] registered [ 0.307137] acpiphp: Slot [26] registered [ 0.308108] acpiphp: Slot [27] registered [ 0.311116] acpiphp: Slot [28] registered [ 0.313151] acpiphp: Slot [29] registered [ 0.314113] acpiphp: Slot [30] registered [ 0.316114] acpiphp: Slot [31] registered [ 0.318259] PCI host bridge to bus 0000:00 [ 0.319022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331432] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334564] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339468] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.356055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363777] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366669] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373963] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402471] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446979] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503042] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517284] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.527028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.536026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562036] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.573400] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.590017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626330] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.646025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.667021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.678828] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.688017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.698015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.730885] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.734457] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.738539] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.741554] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.744507] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.752075] iommu: Default domain type: Passthrough [ 0.754512] SCSI subsystem initialized [ 0.756190] ACPI: bus type USB registered [ 0.758240] usbcore: registered new interface driver usbfs [ 0.760116] usbcore: registered new interface driver hub [ 0.762182] usbcore: registered new device driver usb [ 0.765243] pps_core: LinuxPPS API ver. 1 registered [ 0.767015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.772126] PTP clock support registered [ 0.774063] EDAC MC: Ver: 3.0.0 [ 0.776241] PCI: Using ACPI for IRQ routing [ 0.778521] NetLabel: Initializing [ 0.780013] NetLabel: domain hash size = 128 [ 0.782013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.783116] NetLabel: unlabeled traffic allowed by default [ 0.786139] vgaarb: loaded [ 0.788454] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.790044] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.796458] clocksource: Switched to clocksource kvm-clock [ 0.907561] VFS: Disk quotas dquot_6.6.0 [ 0.909105] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.911524] *** VALIDATE ramfs *** [ 0.912633] *** VALIDATE hugetlbfs *** [ 0.914253] pnp: PnP ACPI init [ 0.916801] pnp: PnP ACPI: found 6 devices [ 0.937457] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.941545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.944258] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.946565] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.949054] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.952211] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.955338] NET: Registered protocol family 2 [ 0.958088] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.963328] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.968305] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.973869] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977677] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980626] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.983857] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.986577] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989620] NET: Registered protocol family 1 [ 0.992330] RPC: Registered named UNIX socket transport module. [ 0.993996] RPC: Registered udp transport module. [ 0.995697] RPC: Registered tcp transport module. [ 0.997025] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.999101] NET: Registered protocol family 44 [ 1.000804] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.002894] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.004965] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.007299] PCI: CLS 0 bytes, default 64 [ 1.009467] Unpacking initramfs... [ 2.694909] debug: unmapping init [mem 0xffff8ebf3cc54000-0xffff8ebf3ffbffff] [ 2.707980] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.711248] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.716583] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.571720] Initialise system trusted keyrings [ 3.575919] Key type blacklist registered [ 3.579574] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.592875] zbud: loaded [ 3.597954] *** VALIDATE nfs *** [ 3.599711] *** VALIDATE nfs4 *** [ 3.601511] pstore: using deflate compression [ 3.607500] Platform Keyring initialized [ 3.910417] NET: Registered protocol family 38 [ 3.916435] Key type asymmetric registered [ 3.921860] Asymmetric key parser 'x509' registered [ 3.924567] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.931527] io scheduler mq-deadline registered [ 3.933821] io scheduler kyber registered [ 3.939434] io scheduler bfq registered [ 3.946526] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.952117] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.955572] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.963861] ACPI: Power Button [PWRF] [ 3.973940] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.984412] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.005706] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.021781] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.052791] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.101781] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.142344] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.147612] Non-volatile memory driver v1.3 [ 4.149363] Linux agpgart interface v0.103 [ 4.196623] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.214372] vda: detected capacity change from 0 to 74694656 [ 4.268804] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.273395] vdb: detected capacity change from 0 to 1073741824 [ 4.321733] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.335134] vdc: detected capacity change from 0 to 2621440000 [ 4.373352] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.391728] vdd: detected capacity change from 0 to 2621440000 [ 4.438437] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.445465] vde: detected capacity change from 0 to 4294967296 [ 4.482547] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.488544] vdf: detected capacity change from 0 to 4294967296 [ 4.506431] libphy: Fixed MDIO Bus: probed [ 4.518448] usbcore: registered new interface driver usbserial_generic [ 4.521973] usbserial: USB Serial support registered for generic [ 4.524560] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.530809] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.533236] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.536123] mousedev: PS/2 mouse device common for all mice [ 4.540420] rtc_cmos 00:05: RTC can wake from S4 [ 4.546572] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.546940] rtc_cmos 00:05: registered as rtc0 [ 4.554391] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.559177] intel_pstate: CPU model not supported [ 4.559804] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.564910] hid: raw HID events driver (C) Jiri Kosina [ 4.566983] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.567824] usbcore: registered new interface driver usbhid [ 4.572772] usbhid: USB HID core driver [ 4.574924] drop_monitor: Initializing network drop monitor service [ 4.578561] Initializing XFRM netlink socket [ 4.581807] NET: Registered protocol family 10 [ 4.586306] Segment Routing with IPv6 [ 4.588205] NET: Registered protocol family 17 [ 4.590764] mpls_gso: MPLS GSO support [ 4.596309] RAS: Correctable Errors collector initialized. [ 4.598948] AVX version of gcm_enc/dec engaged. [ 4.600949] AES CTR mode by8 optimization enabled [ 4.696527] sched_clock: Marking stable (4696429906, 0)->(5703316288, -1006886382) [ 4.701786] registered taskstats version 1 [ 4.704566] Loading compiled-in X.509 certificates [ 4.707370] zswap: loaded using pool lzo/zbud [ 4.740061] Key type big_key registered [ 4.779394] Key type encrypted registered [ 4.782989] ima: No TPM chip found, activating TPM-bypass! [ 4.791597] ima: Allocated hash algorithm: sha1 [ 4.800534] ima: No architecture policies found [ 4.803857] evm: Initialising EVM extended attributes: [ 4.807307] evm: security.selinux [ 4.811244] evm: security.ima [ 4.815425] evm: security.capability [ 4.818262] evm: HMAC attrs: 0x1 [ 4.828042] rtc_cmos 00:05: setting system clock to 2026-08-11 02:19:05 UTC (1786414745) [ 4.845739] debug: unmapping init [mem 0xffffffff86803000-0xffffffff869fffff] [ 4.850562] debug: unmapping init [mem 0xffffffff85582000-0xffffffff85858fff] [ 4.861335] Write protecting the kernel read-only data: 28672k [ 4.873874] debug: unmapping init [mem 0xffffffff83c03000-0xffffffff83dfffff] [ 4.879541] debug: unmapping init [mem 0xffffffff84514000-0xffffffff845fffff] [ 4.977317] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.006744] systemd[1]: Detected virtualization kvm. [ 5.011866] systemd[1]: Detected architecture x86-64. [ 5.019201] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.090412] systemd[1]: No hostname configured. [ 5.092636] systemd[1]: Set hostname to . [ 5.095688] random: systemd: uninitialized urandom read (16 bytes read) [ 5.099434] systemd[1]: Initializing machine ID from random generator. [ 5.250653] random: ln: uninitialized urandom read (6 bytes read) [ 5.484937] random: systemd: uninitialized urandom read (16 bytes read) [ 5.490956] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.496900] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.507525] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.450269] device-mapper: uevent: version 1.0.3 [ 7.452484] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.088795] virtio_net virtio0 ens2: renamed from eth0 [ 9.201851] random: fast init done [ 9.486994] scsi host0: ata_piix [ 9.526402] scsi host1: ata_piix [ 9.527917] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.536722] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.341805] random: crng init done [ 14.343993] random: 7 urandom warning(s) missed due to ratelimiting [ 16.439895] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.296379] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.312285] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.382222] SELinux: Disabled at runtime. [ 22.519031] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.535829] systemd[1]: Detected virtualization kvm. [ 22.538618] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.368913] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.381639] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.393791] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.398477] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.405996] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.433294] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.483503] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 24.853406] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 25.959936] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.987731] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.089256] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.507066] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.622264] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.390257] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.275694] NFS: Registering the id_resolver key type [ 33.278686] Key type id_resolver registered [ 33.281356] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg618-server login: [ 87.102846] hrtimer: interrupt took 6074609 ns [ 99.961316] libcfs: loading out-of-tree module taints kernel. [ 100.024422] Key type ._llcrypt registered [ 100.028691] Key type .llcrypt registered [ 100.186685] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 120.605631] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 122.562419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 122.587579] alg: No test for adler32 (adler32-zlib) [ 124.127523] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 125.172788] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 126.944159] Key type lgssc registered [ 128.429245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.471657] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.681948] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 213.626724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.656835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.883104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.927367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.056757] Lustre: lustre-MDT0000: new disk, initializing [ 215.199110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.225790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.989074] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.624872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.741308] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.778614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.787583] Lustre: Skipped 1 previous similar message [ 239.961401] Lustre: lustre-MDT0001: new disk, initializing [ 240.125479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.183086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 240.213115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.625217] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.287955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 262.171788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.381525] Lustre: lustre-OST0000: new disk, initializing [ 262.385789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 262.394819] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.455744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.719584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.737295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.788490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.398115] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.163613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.262227] Lustre: lustre-OST0001: new disk, initializing [ 284.265416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.271147] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.346340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.630675] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.419909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 293.439883] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 293.469748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 304.772960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 317.024789] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 325.447252] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing check_logdir /tmp/testlogs/ [ 331.413872] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing yml_node [ 336.292721] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 339.368535] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 342.445934] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 344.329636] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 22:24:42 EDT 2026 [ 363.207336] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 364.807263] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 383.457184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 383.468634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.480853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.505446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.509854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.827328] Lustre: server umount lustre-MDT0000 complete [ 391.425174] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415132 with bad export cookie 15102502781872929506 [ 391.432207] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.442119] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 395.745734] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.752040] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.756980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.756988] Lustre: Skipped 3 previous similar messages [ 395.762698] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 395.805222] Lustre: Skipped 3 previous similar messages [ 396.772343] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.796345] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 397.899169] Lustre: server umount lustre-MDT0001 complete [ 402.276263] Lustre: server umount lustre-OST0000 complete [ 407.037717] Lustre: server umount lustre-OST0001 complete [ 415.996967] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 428.625054] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 471.854850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.321951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 472.394453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.610919] Lustre: lustre-MDT0000: new disk, initializing [ 472.735281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.759809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 477.347314] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 488.771439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.938929] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 489.025807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 489.031872] Lustre: Skipped 1 previous similar message [ 489.156584] Lustre: lustre-MDT0001: new disk, initializing [ 489.291738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 489.384813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 489.398735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 494.381739] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 505.146875] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 512.012143] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 519.415752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.809487] Lustre: lustre-OST0000: new disk, initializing [ 519.818714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 519.832806] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 519.943955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.939687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 520.949613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 521.106683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 527.687432] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 537.816267] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 542.884275] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 543.107494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 548.070525] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 548.317776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 551.907662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 551.909587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.928678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.941106] Lustre: Skipped 1 previous similar message [ 551.954540] Lustre: Skipped 4 previous similar messages [ 556.533749] Lustre: server umount lustre-OST0000 complete [ 562.155861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.161400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 562.180791] Lustre: Skipped 1 previous similar message [ 562.208880] Lustre: Skipped 1 previous similar message [ 568.006801] Lustre: server umount lustre-MDT0000 complete [ 571.363099] LustreError: 15966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.383766] LustreError: 15966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 572.020873] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415312 with bad export cookie 15102502781872931039 [ 572.045666] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 572.067046] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.628183] Lustre: server umount lustre-MDT0001 complete [ 577.008616] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:28:35 (1786415315) [ 584.311949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.922607] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.065519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.021333] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 590.317042] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.360965] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 599.523594] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.549548] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 600.063291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.054604] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 616.517276] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 623.180946] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 629.951382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 636.767856] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 645.459513] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 665.062712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.064597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 665.093199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 665.109602] Lustre: Skipped 3 previous similar messages [ 670.612072] Lustre: server umount lustre-OST0000 complete [ 685.542889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.558922] Lustre: Skipped 7 previous similar messages [ 689.121072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 689.390821] Lustre: server umount lustre-MDT0000 complete [ 690.668401] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.701956] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 693.519783] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415434 with bad export cookie 15102502781872931921 [ 693.524655] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.534120] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 694.054761] Lustre: server umount lustre-MDT0001 complete [ 706.625401] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 709.851278] Key type lgssc unregistered [ 710.254929] LNet: 22878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.273444] LNetError: 22878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.298172] LNet: Removed LNI 192.168.206.118@tcp [ 711.421148] Key type .llcrypt unregistered [ 711.431501] Key type ._llcrypt unregistered [ 736.434057] Key type ._llcrypt registered [ 736.439701] Key type .llcrypt registered [ 736.565578] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 738.216707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.306391] alg: No test for adler32 (adler32-zlib) [ 739.501335] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 739.820469] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 741.584189] Key type lgssc registered [ 742.945382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 751.903224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.923633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.584406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.930498] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 763.882127] LustreError: 24293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.900981] LustreError: 24293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 768.945164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.009927] LustreError: 24293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.737220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 774.089990] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 782.248296] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 788.571878] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 796.362449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 796.893597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.982909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 803.226946] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 811.133243] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 828.196875] Lustre: server umount lustre-OST0000 complete [ 836.065730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 836.070630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.086222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 836.093269] Lustre: Skipped 1 previous similar message [ 841.193594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.204590] Lustre: Skipped 2 previous similar messages [ 842.311444] Lustre: server umount lustre-MDT0000 complete [ 845.779335] LustreError: 27517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415586 with bad export cookie 935898227256053646 [ 845.785274] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 845.793377] LustreError: 27517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 856.191526] Lustre: server umount lustre-MDT0001 complete [ 868.324991] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 870.909785] Key type lgssc unregistered [ 871.145742] LNet: 28273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.155836] LNetError: 28273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.178771] LNet: Removed LNI 192.168.206.118@tcp [ 872.200790] Key type .llcrypt unregistered [ 872.203935] Key type ._llcrypt unregistered [ 896.169371] Key type ._llcrypt registered [ 896.171801] Key type .llcrypt registered [ 896.251188] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 897.274988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 897.478407] alg: No test for adler32 (adler32-zlib) [ 898.630148] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 898.892379] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 900.528160] Key type lgssc registered [ 901.786235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 908.848401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 908.883348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.197256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.293561] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 919.527662] LustreError: 29687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.548323] LustreError: 29687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 921.767550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.034292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 926.510488] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 934.262572] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 939.627150] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 951.520381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.773973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.869078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 957.385295] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 964.519471] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 975.330545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 975.337743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.356902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.381466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.395813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.400847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.967876] Lustre: server umount lustre-OST0000 complete [ 987.618983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.624748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 987.628738] Lustre: Skipped 1 previous similar message [ 987.635202] Lustre: Skipped 2 previous similar messages [ 991.634705] Lustre: server umount lustre-MDT0000 complete [ 992.740857] LustreError: 30702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.750650] LustreError: 30702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 995.480813] LustreError: 30421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415736 with bad export cookie 8940129839055634776 [ 995.488264] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.502504] LustreError: 30421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 995.781738] Lustre: server umount lustre-MDT0001 complete [ 1010.224617] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1012.978411] Key type lgssc unregistered [ 1013.336694] LNet: 33565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.342146] LNetError: 33565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.352659] LNet: Removed LNI 192.168.206.118@tcp [ 1014.153423] Key type .llcrypt unregistered [ 1014.155473] Key type ._llcrypt unregistered [ 1038.805849] Key type ._llcrypt registered [ 1038.808763] Key type .llcrypt registered [ 1038.903741] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1039.925795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.077883] alg: No test for adler32 (adler32-zlib) [ 1041.253192] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1041.510163] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1043.225141] Key type lgssc registered [ 1044.332619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1052.002922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1052.024377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.391574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.403905] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1063.912266] LustreError: 34978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.930879] LustreError: 34978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1065.711148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.139417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.583737] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1079.134398] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1085.382885] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1091.689344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1092.029109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.609327] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1099.199817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1106.397431] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.208397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1119.220486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.238541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.742473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.745447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.752552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.775207] Lustre: Skipped 1 previous similar message [ 1125.869205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.880066] Lustre: Skipped 1 previous similar message [ 1130.990745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.016762] Lustre: Skipped 1 previous similar message [ 1132.003036] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1132.280619] Lustre: server umount lustre-OST0000 complete [ 1136.100907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.108806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.121838] Lustre: Skipped 1 previous similar message [ 1136.141638] Lustre: Skipped 1 previous similar message [ 1141.833683] Lustre: server umount lustre-MDT0000 complete [ 1145.656269] LustreError: 37767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415886 with bad export cookie 2896686193711371608 [ 1145.657713] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.677342] LustreError: 37767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1146.087345] Lustre: server umount lustre-MDT0001 complete [ 1158.957966] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1161.579269] Key type lgssc unregistered [ 1161.887426] LNet: 38521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.896804] LNetError: 38521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.920160] LNet: Removed LNI 192.168.206.118@tcp [ 1162.818052] Key type .llcrypt unregistered [ 1162.823854] Key type ._llcrypt unregistered [ 1187.930935] Key type ._llcrypt registered [ 1187.935976] Key type .llcrypt registered [ 1188.025808] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1188.987993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.047254] alg: No test for adler32 (adler32-zlib) [ 1190.041432] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1190.214655] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1191.840206] Key type lgssc registered [ 1192.928785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1200.709386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1200.748415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.317077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.836466] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1211.881691] LustreError: 39939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.903398] LustreError: 39939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1215.161311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.673062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.570427] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1228.686349] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1234.371315] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1240.268436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1240.561670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.598784] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1246.712445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1252.917949] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1259.915694] Lustre: lustre-OST0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 1261.033839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1261.041262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.054057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1262.056344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1262.059307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1262.081099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.012888] Lustre: lustre-OST0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 1265.020336] Lustre: Skipped 1 previous similar message [ 1270.119746] Lustre: lustre-OST0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 1270.130375] Lustre: Skipped 2 previous similar messages [ 1272.296887] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1272.465393] Lustre: server umount lustre-OST0000 complete [ 1292.261724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1292.273133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.297979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.314857] Lustre: Skipped 2 previous similar messages [ 1298.543569] Lustre: server umount lustre-MDT0000 complete [ 1302.780566] LustreError: 39920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416043 with bad export cookie 3575585506970934852 [ 1302.802258] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.805347] LustreError: 39920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1317.344572] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1317.770939] Lustre: server umount lustre-MDT0001 complete [ 1328.939461] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1332.406391] Key type lgssc unregistered [ 1332.838759] LNet: 43651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.847121] LNetError: 43651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.862703] LNet: Removed LNI 192.168.206.118@tcp [ 1333.767511] Key type .llcrypt unregistered [ 1333.769986] Key type ._llcrypt unregistered [ 1358.135055] Key type ._llcrypt registered [ 1358.136746] Key type .llcrypt registered [ 1358.267461] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1359.274663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1359.303686] alg: No test for adler32 (adler32-zlib) [ 1360.290375] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1360.507305] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1362.192400] Key type lgssc registered [ 1363.262422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1370.090082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1370.104474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.591387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.955725] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1381.857661] LustreError: 45075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.879468] LustreError: 45075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1384.133824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.362538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.423299] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1395.776889] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1400.618601] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1406.665188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1406.943768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.900752] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1417.231385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1420.563735] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1428.365803] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 1430.509548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1430.524641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.542076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.546416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.546717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.581338] Lustre: Skipped 2 previous similar messages [ 1433.981595] Lustre: server umount lustre-MDT0000 complete [ 1437.673300] LustreError: 47032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.711415] LustreError: 47032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1438.538800] LustreError: 45053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416179 with bad export cookie 1500645619298833286 [ 1438.561330] LustreError: 45053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1438.578815] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.616802] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.18@tcp (stopping) [ 1438.625564] Lustre: Skipped 1 previous similar message [ 1442.785675] LustreError: 45069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.787130] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.814294] LustreError: 45069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1442.845706] Lustre: Skipped 1 previous similar message [ 1442.864533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1445.229266] Lustre: server umount lustre-MDT0001 complete [ 1466.673845] Lustre: server umount lustre-OST0000 complete [ 1483.205153] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1485.944233] Key type lgssc unregistered [ 1486.187673] LNet: 48935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.198957] LNetError: 48935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.214701] LNet: Removed LNI 192.168.206.118@tcp [ 1487.072544] Key type .llcrypt unregistered [ 1487.078570] Key type ._llcrypt unregistered [ 1513.740750] Key type ._llcrypt registered [ 1513.744702] Key type .llcrypt registered [ 1513.893690] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1514.761983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1514.780178] alg: No test for adler32 (adler32-zlib) [ 1515.916319] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1516.130674] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1517.760619] Key type lgssc registered [ 1518.912744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.292538] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1526.312192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1541.856364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1546.976328] LustreError: 50327:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 1573.602701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.186988] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1588.362264] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1753.820574] Lustre: server umount lustre-OST0000 complete [ 1771.775437] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1774.815519] Key type lgssc unregistered [ 1775.155631] LNet: 52006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.167690] LNetError: 52006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.196145] LNet: Removed LNI 192.168.206.118@tcp [ 1775.975749] Key type .llcrypt unregistered [ 1775.983965] Key type ._llcrypt unregistered [ 1799.327954] Key type ._llcrypt registered [ 1799.331300] Key type .llcrypt registered [ 1799.429559] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1800.365080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1800.408605] alg: No test for adler32 (adler32-zlib) [ 1801.476352] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1801.717284] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1803.352163] Key type lgssc registered [ 1804.230603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.444322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.460592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.882403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.944493] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1823.201879] LustreError: 53421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.221384] LustreError: 53421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1828.296377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.327639] LustreError: 53422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.817172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.842904] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1841.720695] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1848.228639] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.607877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.891228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.973368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1861.119809] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1868.914142] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1873.378056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1873.387380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.397189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1874.417727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.550949] Lustre: server umount lustre-OST0000 complete [ 1884.641371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.659392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.667531] Lustre: Skipped 1 previous similar message [ 1887.757229] Lustre: server umount lustre-MDT0000 complete [ 1889.763817] LustreError: 53422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.784823] LustreError: 53422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1891.474850] LustreError: 53403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416632 with bad export cookie 16927519072853801411 [ 1891.487229] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.495491] LustreError: 53403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.879649] Lustre: server umount lustre-MDT0001 complete [ 1902.437635] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1905.017768] Key type lgssc unregistered [ 1905.278902] LNet: 56954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.292839] LNetError: 56954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.332551] LNet: Removed LNI 192.168.206.118@tcp [ 1906.207440] Key type .llcrypt unregistered [ 1906.213649] Key type ._llcrypt unregistered [ 1930.642334] Key type ._llcrypt registered [ 1930.645358] Key type .llcrypt registered [ 1930.795164] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1932.260485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.311092] alg: No test for adler32 (adler32-zlib) [ 1933.298348] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1933.614814] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1935.400825] Key type lgssc registered [ 1936.543087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.392883] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.408664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.971320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1965.088311] LustreError: 58342:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 1991.776357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.681892] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2005.624685] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2013.098734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.671413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.748884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2018.485187] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2018.790120] LustreError: 58369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.808951] LustreError: 58369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2023.930692] LustreError: 58368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.052499] LustreError: 58369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.088838] LustreError: 58369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2029.376320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.754531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2034.859802] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2044.376195] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2051.336612] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.629266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2054.637881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.652892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.654100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.655194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.673649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.214859] Lustre: server umount lustre-OST0000 complete [ 2081.098158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.102555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.122270] Lustre: Skipped 1 previous similar message [ 2081.138061] Lustre: Skipped 2 previous similar messages [ 2086.369359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.377366] Lustre: Skipped 1 previous similar message [ 2087.468241] Lustre: server umount lustre-MDT0000 complete [ 2091.083515] LustreError: 58350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416831 with bad export cookie 9328179983364251957 [ 2091.092316] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.099627] LustreError: 58350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2097.630558] Lustre: server umount lustre-MDT0001 complete [ 2108.665786] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2111.558518] Key type lgssc unregistered [ 2111.837419] LNet: 62093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.847639] LNetError: 62093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.867753] LNet: Removed LNI 192.168.206.118@tcp [ 2112.947645] Key type .llcrypt unregistered [ 2112.954250] Key type ._llcrypt unregistered [ 2136.611818] Key type ._llcrypt registered [ 2136.616291] Key type .llcrypt registered [ 2136.723439] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2138.074733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.122214] alg: No test for adler32 (adler32-zlib) [ 2139.163821] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2139.435277] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2141.160907] Key type lgssc registered [ 2142.133386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.842678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.890225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.388990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.409415] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2162.670643] LustreError: 63505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.695566] LustreError: 63505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2166.635201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.162176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.851318] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2180.114087] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2187.098655] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2196.015715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.511439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.557048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2203.808380] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2212.692171] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.292729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.305093] Lustre: Skipped 1 previous similar message [ 2232.306284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2232.308073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2232.347320] Lustre: Skipped 1 previous similar message [ 2237.433820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.442563] Lustre: Skipped 1 previous similar message [ 2237.821543] Lustre: server umount lustre-OST0000 complete [ 2242.531682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.534891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.552893] Lustre: Skipped 1 previous similar message [ 2242.567477] Lustre: Skipped 1 previous similar message [ 2247.653787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.668713] Lustre: Skipped 1 previous similar message [ 2252.770732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.784669] Lustre: Skipped 1 previous similar message [ 2256.352534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2256.668087] Lustre: server umount lustre-MDT0000 complete [ 2257.890163] LustreError: 64235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.906580] LustreError: 64235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2260.991066] LustreError: 66109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417001 with bad export cookie 16923424528427915947 [ 2260.995589] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.011449] LustreError: 66109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2261.511318] Lustre: server umount lustre-MDT0001 complete [ 2271.944365] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2274.491309] Key type lgssc unregistered [ 2274.760609] LNet: 67067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.773856] LNetError: 67067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.801332] LNet: Removed LNI 192.168.206.118@tcp [ 2275.630435] Key type .llcrypt unregistered [ 2275.637665] Key type ._llcrypt unregistered [ 2300.122787] Key type ._llcrypt registered [ 2300.125528] Key type .llcrypt registered [ 2300.264509] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2301.345872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.432216] alg: No test for adler32 (adler32-zlib) [ 2302.481587] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2302.688114] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2304.418070] Key type lgssc registered [ 2305.608936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2313.973993] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:57:32 (1786417052) [ 2320.644523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2320.684387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.083576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.558668] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2331.630302] LustreError: 68777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.653471] LustreError: 68777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2334.843502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.340949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2339.587453] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2346.815073] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2352.955135] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2360.266131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2360.856054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2366.984355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2367.281625] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2375.672178] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2385.519256] Lustre: Failing over lustre-MDT0000 [ 2385.783565] Lustre: server umount lustre-MDT0000 complete [ 2386.400990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.408815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.425120] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.910167] LustreError: 71181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.506807] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.980603] LustreError: 68774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.032429] LustreError: 68774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2396.877185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.082071] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.181119] Lustre: *** cfs_fail_loc=135, val=0*** [ 2397.284140] LustreError: 71865:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2397.293221] LustreError: 71865:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.118@tcp: cfg command failed: rc = -2 [ 2397.303216] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2397.303216] [ 2397.309059] LustreError: MGC192.168.206.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2397.325881] LustreError: 71853:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2397.332430] LustreError: 71853:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2397.341571] LustreError: 71853:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2397.352452] LustreError: 71853:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2397.371689] Lustre: server umount lustre-MDT0000 complete [ 2397.375434] LustreError: 71853:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2399.081106] LustreError: 68774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.108683] LustreError: 68774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2405.660652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.905432] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.268372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.343637] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2409.323909] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2411.522873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2411.560842] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2411.624786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2411.840924] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2421.218663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.225562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.239554] Lustre: Skipped 1 previous similar message [ 2421.243954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.000588] Lustre: server umount lustre-OST0000 complete [ 2431.973340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.974731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2431.998346] Lustre: Skipped 1 previous similar message [ 2432.001690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2432.018592] Lustre: Skipped 2 previous similar messages [ 2435.917171] Lustre: server umount lustre-MDT0000 complete [ 2437.088730] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.106663] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2439.743371] LustreError: 68757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417180 with bad export cookie 5532116381702915300 [ 2439.751749] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.065455] Lustre: server umount lustre-MDT0001 complete [ 2451.394384] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2454.393696] Key type lgssc unregistered [ 2454.841966] LNet: 73648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.853705] LNetError: 73648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.871262] LNet: Removed LNI 192.168.206.118@tcp [ 2456.001800] Key type .llcrypt unregistered [ 2456.004232] Key type ._llcrypt unregistered [ 2485.105891] Key type ._llcrypt registered [ 2485.111462] Key type .llcrypt registered [ 2485.237100] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2486.505610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2486.517599] alg: No test for adler32 (adler32-zlib) [ 2487.559255] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2487.807756] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2489.497239] Key type lgssc registered [ 2490.730078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.183200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2498.224334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.696075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.110967] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2510.307603] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.324563] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2514.810649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.314500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.086223] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2529.273115] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.960448] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2541.794378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2542.150745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.199815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2548.760802] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2557.121781] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2564.560239] Lustre: Failing over lustre-MDT0000 [ 2564.581225] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.602776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2564.602967] Lustre: Skipped 1 previous similar message [ 2564.612630] Lustre: Skipped 2 previous similar messages [ 2564.953923] Lustre: server umount lustre-MDT0000 complete [ 2565.001834] LustreError: 75056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.699054] LustreError: 75057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.718053] LustreError: 75057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2574.821115] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.844831] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2576.291140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.474949] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.705778] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2576.831562] LustreError: 78161:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2576.836113] LustreError: 78161:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.118@tcp: cfg command failed: rc = -9 [ 2576.845304] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2576.845304] [ 2576.852322] LustreError: MGC192.168.206.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2576.867711] LustreError: 78149:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2576.875567] LustreError: 78149:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2576.894106] LustreError: 78149:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2576.897489] LustreError: 78149:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2576.939219] Lustre: server umount lustre-MDT0000 complete [ 2576.947196] LustreError: 78149:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2579.941646] LustreError: 75788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.976908] LustreError: 75788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2584.490114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.624578] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.636429] LustreError: 78508:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.118@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000001e72994) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2584.652090] LustreError: 75052:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2584.930681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.997969] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2586.474293] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2589.905674] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2590.199049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2590.247887] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2590.322827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2600.435608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2600.444167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.457032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.494490] Lustre: Skipped 2 previous similar messages [ 2605.540189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.550447] Lustre: Skipped 1 previous similar message [ 2610.676862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.689304] Lustre: Skipped 1 previous similar message [ 2612.192168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2612.380513] Lustre: server umount lustre-OST0000 complete [ 2620.920540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.928277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.969471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.978876] Lustre: Skipped 2 previous similar messages [ 2622.722796] Lustre: server umount lustre-MDT0000 complete [ 2626.027957] LustreError: 75055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.060570] LustreError: 75055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2627.436408] LustreError: 75041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417368 with bad export cookie 12640851107717439127 [ 2627.460499] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2627.878875] Lustre: server umount lustre-MDT0001 complete [ 2640.455327] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2643.303092] Key type lgssc unregistered [ 2643.608130] LNet: 79947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2643.614451] LNetError: 79947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2643.627052] LNet: Removed LNI 192.168.206.118@tcp [ 2644.663459] Key type .llcrypt unregistered [ 2644.666758] Key type ._llcrypt unregistered [ 2659.663597] Key type ._llcrypt registered [ 2659.668323] Key type .llcrypt registered [ 2659.837622] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2660.184602] Key type .llcrypt unregistered [ 2660.188289] Key type ._llcrypt unregistered [ 2688.841465] Key type ._llcrypt registered [ 2688.846389] Key type .llcrypt registered [ 2688.969383] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2690.183748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.212623] alg: No test for adler32 (adler32-zlib) [ 2691.272086] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2691.473636] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2693.128155] Key type lgssc registered [ 2694.493534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2703.119322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2703.181986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.961054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.846460] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2715.106435] LustreError: 81854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.126124] LustreError: 81854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2719.801746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.188744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.167707] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2732.856533] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2739.264234] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2746.580980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.026839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.142660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2753.608441] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2760.996136] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2776.037566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.045911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.073583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.599150] Lustre: server umount lustre-OST0000 complete [ 2781.666561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2781.678361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.693303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2785.250955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2785.252746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.826128] Lustre: server umount lustre-MDT0000 complete [ 2790.369119] LustreError: 82844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.395408] LustreError: 82844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2791.705780] LustreError: 81835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417532 with bad export cookie 10098170623787364626 [ 2791.713791] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2791.721809] LustreError: 81835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2802.490353] Lustre: server umount lustre-MDT0001 complete [ 2815.518692] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2818.550394] Key type lgssc unregistered [ 2818.926407] LNet: 85416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2818.949665] LNetError: 85416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2818.975769] LNet: Removed LNI 192.168.206.118@tcp [ 2819.980916] Key type .llcrypt unregistered [ 2819.987664] Key type ._llcrypt unregistered [ 2841.434870] Key type ._llcrypt registered [ 2841.438216] Key type .llcrypt registered [ 2841.555421] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2842.424318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2842.441830] alg: No test for adler32 (adler32-zlib) [ 2843.624791] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2843.929551] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2845.585791] Key type lgssc registered [ 2846.891056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2854.239555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2854.261897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.666137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.379393] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2866.148938] LustreError: 86809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.172216] LustreError: 86809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2868.656378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.090110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.272569] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2879.878420] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2885.647475] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2892.088205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.545727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.905810] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2900.734139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2906.779681] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2916.322137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2916.335582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.347106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.370324] Lustre: Skipped 1 previous similar message [ 2916.402930] Lustre: Skipped 1 previous similar message [ 2916.553049] Lustre: server umount lustre-OST0000 complete [ 2921.443452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.475042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.493721] Lustre: Skipped 1 previous similar message [ 2926.564453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2926.584702] Lustre: Skipped 1 previous similar message [ 2926.989273] Lustre: server umount lustre-MDT0000 complete [ 2930.862029] LustreError: 86788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417671 with bad export cookie 16042230171041469846 [ 2930.868317] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2930.886587] LustreError: 86788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2931.208104] Lustre: server umount lustre-MDT0001 complete [ 2945.171731] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2948.835884] Key type lgssc unregistered [ 2949.173755] LNet: 90344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2949.184361] LNetError: 90344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2949.231722] LNet: Removed LNI 192.168.206.118@tcp [ 2950.172909] Key type .llcrypt unregistered [ 2950.175415] Key type ._llcrypt unregistered [ 2979.730628] Key type ._llcrypt registered [ 2979.733391] Key type .llcrypt registered [ 2979.897123] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2981.308848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2981.366227] alg: No test for adler32 (adler32-zlib) [ 2982.522825] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2982.881382] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2984.632281] Key type lgssc registered [ 2985.770437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2994.018223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2994.034784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.561337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.162378] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3005.924881] LustreError: 91759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.954373] LustreError: 91759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3009.202920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.687469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3015.256482] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3023.194809] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3028.944650] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3035.689365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3036.101172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.204404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3041.901926] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3048.785398] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3064.818366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3064.819215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.832187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.861286] Lustre: Skipped 1 previous similar message [ 3065.679541] Lustre: server umount lustre-OST0000 complete [ 3069.922220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.935373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.947530] Lustre: Skipped 1 previous similar message [ 3069.969516] Lustre: Skipped 2 previous similar messages [ 3075.042524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.047554] Lustre: Skipped 1 previous similar message [ 3075.346572] Lustre: server umount lustre-MDT0000 complete [ 3079.020501] LustreError: 91739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417819 with bad export cookie 3716699856873174617 [ 3079.024268] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3079.032200] LustreError: 91739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3079.294567] Lustre: server umount lustre-MDT0001 complete [ 3092.924592] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 3095.828553] Key type lgssc unregistered [ 3096.180565] LNet: 95313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3096.195141] LNetError: 95313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3096.214771] LNet: Removed LNI 192.168.206.118@tcp [ 3097.040320] Key type .llcrypt unregistered [ 3097.042270] Key type ._llcrypt unregistered [ 3122.509277] Key type ._llcrypt registered [ 3122.511617] Key type .llcrypt registered [ 3122.627752] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3123.602592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.678431] alg: No test for adler32 (adler32-zlib) [ 3124.815348] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3125.167686] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 3126.856332] Key type lgssc registered [ 3128.336409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3136.235833] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3136.255413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.841370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3156.960302] LustreError: 96701:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 3183.584227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.418662] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3202.717819] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3209.499757] Lustre: server umount lustre-OST0000 complete [ 3218.250335] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:12:37 (1786417957) [ 3224.702171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3225.164052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.414685] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3230.185958] LustreError: 98339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.198367] LustreError: 98339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3235.301882] LustreError: 98340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.172793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.514568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3243.303673] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3251.056847] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.924647] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3263.627265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.950450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.751671] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3270.144800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3276.764262] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.596483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.610830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.624931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.665349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.670778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.678611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.616852] Lustre: server umount lustre-OST0000 complete [ 3303.911544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.916515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.933332] Lustre: Skipped 1 previous similar message [ 3303.949036] Lustre: Skipped 2 previous similar messages [ 3306.482995] Lustre: server umount lustre-MDT0000 complete [ 3309.026467] LustreError: 98336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.054137] LustreError: 98336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3309.724649] LustreError: 98321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418050 with bad export cookie 3751744132580483118 [ 3309.728669] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.732027] LustreError: 98321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3310.067124] Lustre: server umount lustre-MDT0001 complete [ 3320.079242] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 3322.548807] Key type lgssc unregistered [ 3322.798046] LNet: 102045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3322.808638] LNetError: 102045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3322.830313] LNet: Removed LNI 192.168.206.118@tcp [ 3323.734789] Key type .llcrypt unregistered [ 3323.738846] Key type ._llcrypt unregistered [ 3369.925747] Key type ._llcrypt registered [ 3369.930327] Key type .llcrypt registered [ 3370.080248] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3371.065975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3371.096172] alg: No test for adler32 (adler32-zlib) [ 3372.132361] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3372.398677] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 3374.048289] Key type lgssc registered [ 3375.971655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3384.686225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3384.761484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.312406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.282385] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3395.566591] LustreError: 104232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.597717] LustreError: 104232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3400.675624] LustreError: 104233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.307354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.727296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3406.064650] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3413.543897] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3419.897370] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3426.942342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3427.203238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3431.362916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3432.995936] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3441.532225] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3451.874608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.888880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3456.997654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.018609] Lustre: Skipped 2 previous similar messages [ 3457.633981] Lustre: server umount lustre-OST0000 complete [ 3462.114332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.118155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.125022] Lustre: Skipped 2 previous similar messages [ 3462.135059] Lustre: Skipped 1 previous similar message [ 3467.234649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.241808] Lustre: Skipped 1 previous similar message [ 3467.755479] Lustre: server umount lustre-MDT0000 complete [ 3471.852432] LustreError: 104211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418212 with bad export cookie 4878864263971558521 [ 3471.853371] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3471.862149] LustreError: 104211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3472.341400] Lustre: server umount lustre-MDT0001 complete [ 3482.724730] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 3485.178776] Key type lgssc unregistered [ 3485.434894] LNet: 107772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3485.446262] LNetError: 107772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3485.465619] LNet: Removed LNI 192.168.206.118@tcp [ 3486.185382] Key type .llcrypt unregistered [ 3486.187855] Key type ._llcrypt unregistered [ 3501.862806] Key type ._llcrypt registered [ 3501.866251] Key type .llcrypt registered [ 3501.962950] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3502.868168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3502.979322] alg: No test for adler32 (adler32-zlib) [ 3503.985262] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3504.297790] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 3505.952978] Key type lgssc registered [ 3507.058561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3514.328748] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3514.363539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.017309] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3535.136250] LustreError: 109006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 3561.824185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.972322] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3575.346515] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3582.164193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.470840] LustreError: MGC192.168.206.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3582.485960] LustreError: 110029:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3582.493863] LustreError: 110029:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3582.502996] LustreError: 110029:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3582.639297] Lustre: server umount lustre-MDT0000 complete [ 3582.643819] LustreError: 110029:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3600.443460] Lustre: server umount lustre-OST0000 complete [ 3610.120868] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 3619.066172] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3654.920782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.175118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3655.196460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3655.261357] Lustre: lustre-MDT0000: new disk, initializing [ 3655.315862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.336711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3660.127975] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3671.483845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3671.675423] Lustre: 112819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3671.792434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3671.796381] Lustre: Skipped 1 previous similar message [ 3671.954773] Lustre: lustre-MDT0001: new disk, initializing [ 3672.106160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3672.141724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3672.160807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3676.905749] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3686.391814] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3692.878484] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3699.094573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3699.365414] Lustre: lustre-OST0000: new disk, initializing [ 3699.373670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3699.380332] Lustre: 114788:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3699.517897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.713977] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3705.920848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3705.940953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3706.099804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3713.522828] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3717.701774] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3717.863663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3722.583572] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3722.800704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3726.817738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3726.821612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.826906] LustreError: Skipped 1 previous similar message [ 3726.829632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.847086] Lustre: Skipped 1 previous similar message [ 3730.944728] Lustre: server umount lustre-OST0000 complete [ 3737.063893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.066019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.079854] Lustre: Skipped 1 previous similar message [ 3737.090584] Lustre: Skipped 1 previous similar message [ 3741.192899] Lustre: server umount lustre-MDT0000 complete [ 3744.609642] LustreError: 114787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418485 with bad export cookie 11604553730732816204 [ 3744.615822] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.623206] LustreError: 114787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3745.131647] Lustre: server umount lustre-MDT0001 complete [ 3753.461364] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:21:32 (1786418492) [ 3756.297327] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3769.867805] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 3777.581423] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3805.681765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.894769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3805.920296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3805.985573] Lustre: lustre-MDT0000: new disk, initializing [ 3806.074617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.088213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3809.929305] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3820.187520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.363760] Lustre: 119690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3820.430202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3820.435272] Lustre: Skipped 1 previous similar message [ 3820.547244] Lustre: lustre-MDT0001: new disk, initializing [ 3820.679444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3820.723120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3820.738253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3825.321479] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3833.968621] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3839.161176] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3844.601458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.798294] Lustre: lustre-OST0000: new disk, initializing [ 3844.801733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3844.806633] Lustre: 121656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3844.873461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.233476] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3853.350798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3853.370168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3853.470863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3857.040293] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3861.079586] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3861.240827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3865.651525] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3865.840429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3869.154508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3869.156389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.162793] LustreError: Skipped 1 previous similar message [ 3869.166901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3869.179089] Lustre: Skipped 1 previous similar message [ 3869.193658] Lustre: Skipped 1 previous similar message [ 3873.537956] Lustre: server umount lustre-OST0000 complete [ 3877.345681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3877.355848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.375860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3882.780520] Lustre: server umount lustre-MDT0000 complete [ 3884.515283] LustreError: 120498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.537759] LustreError: 120498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3886.379737] LustreError: 119680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418627 with bad export cookie 11604553730732817086 [ 3886.382809] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3886.403480] LustreError: 119680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3886.743562] Lustre: server umount lustre-MDT0001 complete [ 3894.153925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.546605] LustreError: 123872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.604654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.562426] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3899.889650] LustreError: 123873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.016284] LustreError: 123872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.873145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.828884] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3917.671762] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3922.865270] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3929.338990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.523967] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3944.518074] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3951.268235] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3957.233632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.242628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.253835] Lustre: Skipped 1 previous similar message [ 3957.262575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.267948] Lustre: Skipped 2 previous similar messages [ 3962.372697] Lustre: server umount lustre-OST0000 complete [ 3968.487605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3968.493186] LustreError: Skipped 1 previous similar message [ 3968.506852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.525375] Lustre: Skipped 1 previous similar message [ 3968.534421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.541503] Lustre: Skipped 2 previous similar messages [ 3972.838900] Lustre: server umount lustre-MDT0000 complete [ 3975.650296] LustreError: 123872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.375584] LustreError: 124607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418717 with bad export cookie 11604553730732817968 [ 3976.387954] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3976.392260] LustreError: 124607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3976.767837] Lustre: server umount lustre-MDT0001 complete [ 3987.498364] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 3990.233183] Key type lgssc unregistered [ 3990.644324] LNet: 127503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3990.671086] LNetError: 127503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3990.708171] LNet: Removed LNI 192.168.206.118@tcp [ 3991.552101] Key type .llcrypt unregistered [ 3991.557996] Key type ._llcrypt unregistered [ 4009.318683] Key type ._llcrypt registered [ 4009.323601] Key type .llcrypt registered [ 4009.471737] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 4026.539117] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4028.008503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4028.021132] alg: No test for adler32 (adler32-zlib) [ 4029.289686] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 4029.646480] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 4031.426906] Key type lgssc registered [ 4033.049418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4072.440494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4072.521810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.878320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4073.929518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4074.080600] Lustre: lustre-MDT0000: new disk, initializing [ 4074.224925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.256848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4079.502484] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4092.029933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.220689] Lustre: 130660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4092.341189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4092.346691] Lustre: Skipped 1 previous similar message [ 4092.437853] Lustre: lustre-MDT0001: new disk, initializing [ 4092.586659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4092.622964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4092.640975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4097.896992] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4109.579171] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4115.903272] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4122.995809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.363651] Lustre: lustre-OST0000: new disk, initializing [ 4123.370362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4123.376724] Lustre: 132626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4123.491590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.087779] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4132.906644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4132.925257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4133.029745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4138.892161] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4142.687900] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4142.896324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4147.539530] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4147.781649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4153.825211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4153.831713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.847908] LustreError: Skipped 1 previous similar message [ 4153.848922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4153.865993] Lustre: Skipped 1 previous similar message [ 4155.742375] Lustre: server umount lustre-OST0000 complete [ 4159.458135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4159.463890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.482434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4159.494911] Lustre: Skipped 1 previous similar message [ 4164.065368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4164.068365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.312414] Lustre: server umount lustre-MDT0000 complete [ 4168.940272] LustreError: 130651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418909 with bad export cookie 5042887262522928004 [ 4168.951758] LustreError: 130651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4168.954470] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4169.268105] Lustre: server umount lustre-MDT0001 complete [ 4178.031786] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:28:36 (1786418916) [ 4186.427110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.128140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.238252] LustreError: 135038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.266327] LustreError: 135038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4192.420581] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4197.348428] LustreError: 135037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.466577] LustreError: 135038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.852221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.494191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.563099] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4218.447107] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4224.893194] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4229.863832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.879751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4229.892107] Lustre: Skipped 2 previous similar messages [ 4235.237114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.376122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4241.529037] Lustre: server umount lustre-MDT0000 complete [ 4245.043467] LustreError: 135019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418985 with bad export cookie 5042887262522928886 [ 4245.052845] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.058460] LustreError: 135019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4245.427276] Lustre: server umount lustre-MDT0001 complete [ 4253.690429] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:29:52 (1786418992) [ 4259.802857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4275.361107] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4280.482312] LustreError: 137452:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 4306.272872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.729135] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4321.235331] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4323.749535] Lustre: server umount lustre-OST0000 complete [ 4332.360709] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:31:11 (1786419071) [ 4339.901652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.339330] LustreError: 138901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.451982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.835829] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4345.832396] LustreError: 138902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.906061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.990805] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4368.309657] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4374.473950] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4382.137136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.472818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.485514] Lustre: Skipped 1 previous similar message [ 4389.028853] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4397.541926] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4409.314404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.324806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4409.350341] Lustre: Skipped 3 previous similar messages [ 4409.368833] Lustre: Skipped 4 previous similar messages [ 4412.690120] Lustre: server umount lustre-MDT0000 complete [ 4414.433320] LustreError: 139849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.468691] LustreError: 139849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4417.170125] LustreError: 138883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419157 with bad export cookie 5042887262522929551 [ 4417.189950] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.193977] LustreError: 138883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4417.478348] Lustre: server umount lustre-MDT0001 complete [ 4432.505148] Lustre: server umount lustre-OST0000 complete [ 4442.170923] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:33:00 (1786419180) [ 4449.355049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.968855] LustreError: 142556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.048499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.732263] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4465.025557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.286922] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4479.257732] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4485.775464] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4492.841154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4495.309763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4499.851544] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4507.605864] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4512.556399] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4512.758152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4517.395953] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4517.613292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4520.930203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4520.950830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.975644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4520.988432] Lustre: Skipped 1 previous similar message [ 4525.027470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4534.241447] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4534.592316] Lustre: server umount lustre-OST0000 complete [ 4540.388940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.401993] Lustre: Skipped 1 previous similar message [ 4545.040570] Lustre: server umount lustre-MDT0000 complete [ 4545.507125] LustreError: 142556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.544321] LustreError: 142556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4549.817158] LustreError: 142536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419290 with bad export cookie 5042887262522930622 [ 4549.819036] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4549.832357] LustreError: 142536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4550.531349] Lustre: server umount lustre-MDT0001 complete [ 4561.250925] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:34:59 (1786419299) [ 4568.721224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.288689] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4589.409507] LustreError: 146716:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 4615.136460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.148018] Lustre: Skipped 2 previous similar messages [ 4621.785989] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4628.909802] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4636.265939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.690302] LustreError: 146741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.794974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4641.305777] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4650.321039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.975949] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4662.271627] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4668.020638] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4672.817261] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4672.999470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4676.643618] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4676.851758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4681.705109] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.714768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.724046] Lustre: Skipped 2 previous similar messages [ 4681.724865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4681.735463] Lustre: Skipped 7 previous similar messages [ 4684.724938] Lustre: server umount lustre-MDT0000 complete [ 4688.397623] LustreError: 146724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419429 with bad export cookie 5042887262522931644 [ 4688.403377] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4688.408397] LustreError: 146724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4688.663466] Lustre: server umount lustre-MDT0001 complete [ 4701.918107] Lustre: server umount lustre-OST0000 complete [ 4713.204091] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:37:31 (1786419451) [ 4719.619182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4735.200594] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4740.320526] LustreError: 150913:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 4772.102469] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4779.759700] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4786.907897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.214578] LustreError: 150938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4787.231944] LustreError: 150938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4787.344222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4791.879868] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4801.097574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.484780] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4813.835716] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4819.092431] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.545174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.673573] Lustre: lustre-OST0001: new disk, initializing [ 4825.679224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4825.688288] Lustre: 153781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4831.295131] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4833.338719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4833.346098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4833.443239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4840.369415] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4844.878324] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4845.145452] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4850.858307] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4851.129502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4853.730401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4853.749108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.779938] Lustre: Skipped 1 previous similar message [ 4853.797328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4853.803839] Lustre: Skipped 1 previous similar message [ 4859.757101] Lustre: server umount lustre-OST0000 complete [ 4879.328383] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4879.455896] Lustre: server umount lustre-OST0001 complete [ 4883.426954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4883.436522] LustreError: Skipped 2 previous similar messages [ 4889.543335] Lustre: server umount lustre-MDT0000 complete [ 4893.147812] LustreError: 150920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419633 with bad export cookie 5042887262522932596 [ 4893.150926] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.160262] LustreError: 150920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4893.572178] Lustre: server umount lustre-MDT0001 complete [ 4919.508243] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:40:58 (1786419658) [ 4921.184638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4923.207217] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4924.909757] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:41:03 (1786419663) [ 4931.348116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.659834] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4931.693061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4931.874927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.882343] Lustre: Skipped 6 previous similar messages [ 4935.879126] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4944.188737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.310687] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4944.344989] Lustre: 157344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4948.993768] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4956.668658] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4962.138144] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4969.259429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4969.497466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4976.435592] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4977.848917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4983.810932] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4988.130772] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4988.361798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4992.865658] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4993.057666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4998.630371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4998.633778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4998.645042] Lustre: Skipped 5 previous similar messages [ 4998.651308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4998.672583] Lustre: Skipped 12 previous similar messages [ 5009.376178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5009.568360] Lustre: server umount lustre-OST0000 complete [ 5031.680782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5037.423490] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5039.149853] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.159156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.178427] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5039.187151] Lustre: Skipped 1 previous similar message [ 5044.120916] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5049.223654] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5049.460492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5053.660630] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5053.838636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5058.414386] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91bf847e1800.ost_server_uuid 50 [ 5059.721636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91bf847e1800.ost_server_uuid in FULL state after 0 sec [ 5059.912202] LustreError: 157350:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.18@tcp) returned error from blocking AST (req@0000000017c0c343 x1873195848437376 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004c663a86/0x45fbef3f350586ce lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.18@tcp remote: 0x477b448ecfec2ed9 expref: 5 pid: 158916 timeout: 5159 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5059.930360] LustreError: lustre-MDT0000: A client on nid 192.168.206.18@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5059.936628] LustreError: 157340:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.18@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004c663a86/0x45fbef3f350586ce lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.18@tcp remote: 0x477b448ecfec2ed9 expref: 6 pid: 158916 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5064.677723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5064.691763] LustreError: Skipped 1 previous similar message [ 5070.248189] Lustre: server umount lustre-OST0000 complete [ 5080.522443] Lustre: server umount lustre-MDT0000 complete [ 5084.004686] LustreError: 157335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419824 with bad export cookie 5042887262522933779 [ 5084.011503] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5084.020136] LustreError: 157335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5090.403793] Lustre: server umount lustre-MDT0001 complete [ 5100.207776] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5103.109317] Key type lgssc unregistered [ 5103.372164] LNet: 163467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5103.380905] LNetError: 163467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5104.422697] LNet: Removed LNI 192.168.206.118@tcp [ 5105.500138] Key type .llcrypt unregistered [ 5105.502657] Key type ._llcrypt unregistered [ 5128.128435] Key type ._llcrypt registered [ 5128.129989] Key type .llcrypt registered [ 5128.213762] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5129.058158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5129.194214] alg: No test for adler32 (adler32-zlib) [ 5130.201513] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5130.406035] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5132.088151] Key type lgssc registered [ 5133.227631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5140.785890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5140.836549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.367139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.539307] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5151.725461] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.757342] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5154.893140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.330431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5159.164406] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5166.467801] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.093287] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5177.224752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5177.541434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.460926] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5186.737173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5189.870362] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5195.775065] Lustre: Failing over lustre-MDT0000 [ 5196.027313] Lustre: server umount lustre-MDT0000 complete [ 5196.258941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5196.266817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.281581] LustreError: 164880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.284264] LustreError: 164881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.292093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.320329] Lustre: Skipped 1 previous similar message [ 5202.403044] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.418522] LustreError: 164875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5207.525281] LustreError: 165609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.539321] LustreError: 165609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5207.861081] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.18@tcp arrived at 1786419948 with bad export cookie 4161283183872626496 [ 5207.873522] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5212.641210] Lustre: 164191:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419937/real 1786419937] req@000000004d4b97ae x1873197002679168/t0(0) o400->MGC192.168.206.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786419953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5212.666522] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5213.592703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.762394] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.774213] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5222.884147] LustreError: 164189:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ddab80fc x1873197002683904/t0(0) o250->MGC192.168.206.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5223.128260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.173787] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5226.884362] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5228.522073] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5228.548220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5229.434747] Lustre: lustre-MDT0000: Denying connection for new client 90e8cac1-d3b4-4e9d-81e1-b517efc891ed (at 192.168.206.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5238.762652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.810902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5238.819819] Lustre: Skipped 1 previous similar message [ 5243.883197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5243.903569] Lustre: Skipped 1 previous similar message [ 5244.650623] Lustre: server umount lustre-OST0000 complete [ 5248.826658] LustreError: 168610:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5248.837713] Lustre: 167960:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5248.845451] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5248.876710] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5248.888257] LustreError: 167960:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003a82689b x1873197002698752/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5248.910735] LustreError: 167960:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5248.928848] LustreError: 167960:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5248.940494] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5248.997117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5249.027711] Lustre: Skipped 1 previous similar message [ 5249.042309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5249.409322] Lustre: server umount lustre-MDT0000 complete [ 5253.606973] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419994 with bad export cookie 4161283183872626594 [ 5253.618064] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5253.628961] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5253.642400] LustreError: 164189:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5256.025695] Lustre: server umount lustre-MDT0001 complete [ 5265.924101] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5268.185770] Key type lgssc unregistered [ 5268.438331] LNet: 169365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5268.444975] LNetError: 169365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5268.456652] LNet: Removed LNI 192.168.206.118@tcp [ 5269.189569] Key type .llcrypt unregistered [ 5269.193589] Key type ._llcrypt unregistered [ 5289.283535] Key type ._llcrypt registered [ 5289.286138] Key type .llcrypt registered [ 5289.374038] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5290.184357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5290.392543] alg: No test for adler32 (adler32-zlib) [ 5291.480094] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5291.700945] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5293.360274] Key type lgssc registered [ 5294.583568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5303.201875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5303.245463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.751498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.924186] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5315.044084] LustreError: 170776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.062525] LustreError: 170776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5316.227734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.654496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5321.772599] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5328.873310] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.121487] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5342.323364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.714101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5349.084457] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5349.897556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5356.589623] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5363.119463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.123376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.283818] Lustre: server umount lustre-OST0000 complete [ 5372.897947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5372.916626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.938783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5372.944206] Lustre: Skipped 1 previous similar message [ 5378.510638] Lustre: server umount lustre-MDT0000 complete [ 5378.532301] LustreError: 170772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.558441] LustreError: 170772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5382.313247] LustreError: 170757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420122 with bad export cookie 5131816607038866242 [ 5382.314053] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5382.320030] LustreError: 170757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5382.761468] Lustre: server umount lustre-MDT0001 complete [ 5394.313401] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5397.917588] Key type lgssc unregistered [ 5398.302890] LNet: 174311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5398.312604] LNetError: 174311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5398.338607] LNet: Removed LNI 192.168.206.118@tcp [ 5399.526537] Key type .llcrypt unregistered [ 5399.531177] Key type ._llcrypt unregistered [ 5437.971584] Key type ._llcrypt registered [ 5437.975710] Key type .llcrypt registered [ 5438.062718] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5439.137973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5439.195282] alg: No test for adler32 (adler32-zlib) [ 5440.500990] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5440.894915] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5442.624456] Key type lgssc registered [ 5443.810570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5451.086383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5451.113385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.543247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.924818] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5463.013403] LustreError: 176018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.044200] LustreError: 176018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5465.521359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.816097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5470.152166] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5477.882979] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5483.622632] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5490.300459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5490.644650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.208572] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5497.857490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5505.254331] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5513.186478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.187343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5513.195215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5513.201461] Lustre: Skipped 1 previous similar message [ 5518.239858] Lustre: server umount lustre-OST0000 complete [ 5522.403793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5522.423249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.469441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5522.475479] Lustre: Skipped 1 previous similar message [ 5523.451153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.568721] LustreError: 177046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.591035] LustreError: 177046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5528.763051] Lustre: server umount lustre-MDT0000 complete [ 5532.269488] LustreError: 175999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420272 with bad export cookie 5481903504378029288 [ 5532.286106] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.302261] LustreError: 175999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5532.729223] Lustre: server umount lustre-MDT0001 complete [ 5542.586379] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5544.933957] Key type lgssc unregistered [ 5545.221966] LNet: 179554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5545.230388] LNetError: 179554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5546.311837] LNet: Removed LNI 192.168.206.118@tcp [ 5547.063283] Key type .llcrypt unregistered [ 5547.065296] Key type ._llcrypt unregistered [ 5567.956535] Key type ._llcrypt registered [ 5567.959358] Key type .llcrypt registered [ 5568.072765] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5568.821278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5568.967203] alg: No test for adler32 (adler32-zlib) [ 5570.002566] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5570.207558] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5571.888229] Key type lgssc registered [ 5572.903537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5582.290261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5582.303379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.697069] Lustre: *** cfs_fail_loc=135, val=0*** [ 5583.851964] LustreError: 181077:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5583.862178] LustreError: 181077:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.118@tcp: cfg command failed: rc = -2 [ 5583.871906] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5583.871906] [ 5583.881600] LustreError: MGC192.168.206.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5583.901670] LustreError: 181038:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5583.908509] LustreError: 181038:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5583.916383] LustreError: 181038:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5583.924448] LustreError: 181038:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5584.006825] Lustre: server umount lustre-MDT0000 complete [ 5584.009977] LustreError: 181038:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5590.298341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.676844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.037093] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5601.252702] LustreError: 181449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.267677] LustreError: 181449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5602.934685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.259527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5607.109519] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5614.067604] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5618.500916] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5620.707083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.711375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5620.715809] Lustre: Skipped 1 previous similar message [ 5620.725926] Lustre: Skipped 1 previous similar message [ 5625.827015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5625.837505] Lustre: Skipped 1 previous similar message [ 5626.235823] Lustre: server umount lustre-MDT0000 complete [ 5627.879947] LustreError: 181448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.897351] LustreError: 181448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5629.295184] LustreError: 183015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420369 with bad export cookie 15338912405415970874 [ 5629.296355] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.303504] LustreError: 183015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5629.703110] Lustre: server umount lustre-MDT0001 complete [ 5639.238869] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5641.761823] Key type lgssc unregistered [ 5642.050671] LNet: 183771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.064692] LNetError: 183771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5642.081467] LNet: Removed LNI 192.168.206.118@tcp [ 5642.847664] Key type .llcrypt unregistered [ 5642.849817] Key type ._llcrypt unregistered [ 5664.476396] Key type ._llcrypt registered [ 5664.478404] Key type .llcrypt registered [ 5664.578313] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5664.887444] Key type .llcrypt unregistered [ 5664.893934] Key type ._llcrypt unregistered [ 5681.345872] Key type ._llcrypt registered [ 5681.348779] Key type .llcrypt registered [ 5681.496218] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5682.628097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5682.658021] alg: No test for adler32 (adler32-zlib) [ 5683.844109] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5684.110364] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5685.776171] Key type lgssc registered [ 5686.812187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5694.432626] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5694.477423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5709.984615] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5715.104323] LustreError: 185909:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 5741.729694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.219450] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5753.988614] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5760.638175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.204580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.282361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5765.518942] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5766.638166] LustreError: 185935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.671993] LustreError: 185935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5771.760061] LustreError: 185934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.723713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.065743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5778.008076] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5784.740665] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5791.077389] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5793.953887] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5802.474658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5802.488517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.498370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.100051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.105924] Lustre: Skipped 1 previous similar message [ 5808.518055] Lustre: server umount lustre-OST0000 complete [ 5813.219431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.221671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.239617] Lustre: Skipped 2 previous similar messages [ 5813.251263] Lustre: Skipped 1 previous similar message [ 5818.340410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.349475] Lustre: Skipped 1 previous similar message [ 5820.886863] Lustre: server umount lustre-MDT0000 complete [ 5823.458640] LustreError: 186946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5823.482594] LustreError: 186946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5825.602294] LustreError: 188570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420566 with bad export cookie 16065973826438998826 [ 5825.605826] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5825.609311] LustreError: 188570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5825.827734] Lustre: server umount lustre-MDT0001 complete [ 5838.956928] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 5842.419949] Key type lgssc unregistered [ 5842.872215] LNet: 189831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5842.882079] LNetError: 189831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5842.916505] LNet: Removed LNI 192.168.206.118@tcp [ 5843.791370] Key type .llcrypt unregistered [ 5843.795054] Key type ._llcrypt unregistered [ 5868.248130] Key type ._llcrypt registered [ 5868.254986] Key type .llcrypt registered [ 5868.435579] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 5869.491217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5869.556713] alg: No test for adler32 (adler32-zlib) [ 5870.731498] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5871.090941] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 5872.816204] Key type lgssc registered [ 5873.996312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5881.989526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5882.026479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.644835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.699507] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5894.115572] LustreError: 191244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.145761] LustreError: 191244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5896.635947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.109257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5902.104555] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5910.791612] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5917.988326] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5926.742021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5927.093620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5929.157783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5933.813224] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5942.675237] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5948.561320] Lustre: Failing over lustre-MDT0000 [ 5948.862594] Lustre: server umount lustre-MDT0000 complete [ 5950.878708] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.979732] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.984560] LustreError: 191243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.998287] Lustre: Skipped 1 previous similar message [ 5952.037332] LustreError: 191243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5955.946428] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.064845] LustreError: 191239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.086345] LustreError: 191239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5967.329076] Lustre: 190557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786420692/real 1786420692] req@00000000823f62ed x1873197778629248/t0(0) o400->MGC192.168.206.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786420708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5967.362862] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.667753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.300450] LustreError: 191238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.337430] LustreError: 191238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5977.576838] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x74a4ef9850d1a8bf [ 5977.592939] Lustre: MGC192.168.206.118@tcp: Connection restored to 0@lo (at 0@lo) [ 5977.924920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5977.985749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5982.513387] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 5982.578320] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5983.218581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5983.245122] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5983.309901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5987.069849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5989.145442] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6003.169827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6003.176993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.189620] LustreError: Skipped 1 previous similar message [ 6003.195218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.219432] Lustre: Skipped 2 previous similar messages [ 6003.239769] Lustre: Skipped 1 previous similar message [ 6008.815670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6008.825102] Lustre: Skipped 1 previous similar message [ 6009.384324] Lustre: server umount lustre-OST0000 complete [ 6013.925853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.928994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6013.939759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.042432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.051559] Lustre: Skipped 2 previous similar messages [ 6019.644182] Lustre: server umount lustre-MDT0000 complete [ 6023.102941] LustreError: 194925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420763 with bad export cookie 8405106242050173119 [ 6023.107960] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6023.110260] LustreError: 194925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6023.459800] Lustre: server umount lustre-MDT0001 complete [ 6034.705092] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6037.705835] Key type lgssc unregistered [ 6038.041845] LNet: 196011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6038.055049] LNetError: 196011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6038.077922] LNet: Removed LNI 192.168.206.118@tcp [ 6038.938318] Key type .llcrypt unregistered [ 6038.942913] Key type ._llcrypt unregistered [ 6062.800875] Key type ._llcrypt registered [ 6062.803771] Key type .llcrypt registered [ 6062.941727] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6064.152936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6064.340242] alg: No test for adler32 (adler32-zlib) [ 6065.479693] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6065.776522] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6067.472217] Key type lgssc registered [ 6068.785545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6075.845882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6075.867296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.317694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.548108] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6086.630902] LustreError: 197440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.658949] LustreError: 197440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6090.424930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.782877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6095.079934] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6101.678531] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6106.879198] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6113.180767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6113.541519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.605148] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6121.797129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6128.997813] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6132.972990] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6140.799064] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6162.063713] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6173.166125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6173.172752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.176101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.196654] Lustre: Skipped 1 previous similar message [ 6178.275711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6178.283281] Lustre: Skipped 2 previous similar messages [ 6179.169766] Lustre: server umount lustre-OST0000 complete [ 6182.916450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6182.931687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.958283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6188.516709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6188.518797] Lustre: Skipped 2 previous similar messages [ 6189.290780] Lustre: server umount lustre-MDT0000 complete [ 6192.705189] LustreError: 200264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420933 with bad export cookie 15503064438821488237 [ 6192.713470] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6192.721363] LustreError: 200264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6193.190570] Lustre: server umount lustre-MDT0001 complete [ 6205.214710] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6208.043831] Key type lgssc unregistered [ 6208.479086] LNet: 201325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6208.488295] LNetError: 201325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6208.508292] LNet: Removed LNI 192.168.206.118@tcp [ 6209.457884] Key type .llcrypt unregistered [ 6209.459976] Key type ._llcrypt unregistered [ 6230.022268] Key type ._llcrypt registered [ 6230.024018] Key type .llcrypt registered [ 6230.164347] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6232.024550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6232.047360] alg: No test for adler32 (adler32-zlib) [ 6233.124733] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6233.412558] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6235.128164] Key type lgssc registered [ 6236.349788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6244.103462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6244.124952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.530218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6250.608379] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6256.106466] LustreError: 202593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.133062] LustreError: 202593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6260.491364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.128513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6266.652171] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6275.217580] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6280.330578] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6287.166033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.511732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6290.622749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6294.010635] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6301.955510] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6307.169498] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6316.667968] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6323.701622] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6330.098275] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6342.123569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.129544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6342.132742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6342.160870] Lustre: Skipped 1 previous similar message [ 6342.225452] LustreError: Skipped 1 previous similar message [ 6347.245794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6347.258988] Lustre: Skipped 1 previous similar message [ 6352.355307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6352.358672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.382224] Lustre: Skipped 2 previous similar messages [ 6352.501737] Lustre: server umount lustre-OST0000 complete [ 6357.474343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.477962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.489967] Lustre: Skipped 1 previous similar message [ 6357.499215] Lustre: Skipped 1 previous similar message [ 6362.596191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6362.601226] Lustre: Skipped 1 previous similar message [ 6362.826327] Lustre: server umount lustre-MDT0000 complete [ 6366.662270] LustreError: 205620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421107 with bad export cookie 13275536528349897122 [ 6366.666154] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6366.676019] LustreError: 205620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6367.158236] Lustre: server umount lustre-MDT0001 complete [ 6378.265828] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6380.762695] Key type lgssc unregistered [ 6381.076662] LNet: 207327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6381.097660] LNetError: 207327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6381.111509] LNet: Removed LNI 192.168.206.118@tcp [ 6381.996908] Key type .llcrypt unregistered [ 6382.001631] Key type ._llcrypt unregistered [ 6404.256715] Key type ._llcrypt registered [ 6404.272421] Key type .llcrypt registered [ 6404.399988] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6405.248090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6405.346619] alg: No test for adler32 (adler32-zlib) [ 6406.524454] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6406.862440] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6408.560946] Key type lgssc registered [ 6410.004545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6418.132201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6418.155581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.692718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6424.363471] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6430.180124] LustreError: 208721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.197789] LustreError: 208721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6432.729979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.292657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6437.115306] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6444.253663] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6450.164364] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6456.623967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.910174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6462.288593] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6465.143469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6469.114133] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6483.944623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6483.948394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.962578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6485.989312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6485.990523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6485.995684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.517624] Lustre: server umount lustre-OST0000 complete [ 6491.107881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.130881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6491.143554] Lustre: Skipped 2 previous similar messages [ 6496.227235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.236168] Lustre: Skipped 1 previous similar message [ 6496.510722] Lustre: server umount lustre-MDT0000 complete [ 6500.023536] LustreError: 208700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421240 with bad export cookie 18257340307952892676 [ 6500.025506] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6500.035868] LustreError: 208700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6500.404847] Lustre: server umount lustre-MDT0001 complete [ 6510.712378] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6513.432946] Key type lgssc unregistered [ 6513.695552] LNet: 212407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6513.701462] LNetError: 212407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6513.717196] LNet: Removed LNI 192.168.206.118@tcp [ 6514.516499] Key type .llcrypt unregistered [ 6514.519524] Key type ._llcrypt unregistered [ 6530.405945] Key type ._llcrypt registered [ 6530.408524] Key type .llcrypt registered [ 6530.522374] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6531.484106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6531.506040] alg: No test for adler32 (adler32-zlib) [ 6532.528112] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6532.778430] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6534.465199] Key type lgssc registered [ 6535.490485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6542.144417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6542.167583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.572356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.202553] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6554.084086] LustreError: 213652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6554.086970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.095368] LustreError: 213652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6554.397707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6558.635860] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6566.139374] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6571.864423] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.927254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6578.352687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.961562] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6588.607968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6591.329652] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6604.261110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.265068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6604.286856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6604.306571] Lustre: Skipped 1 previous similar message [ 6609.385534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.390141] Lustre: Skipped 1 previous similar message [ 6610.366214] Lustre: server umount lustre-OST0000 complete [ 6614.497799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6614.509402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6614.524847] Lustre: Skipped 2 previous similar messages [ 6614.535339] Lustre: Skipped 1 previous similar message [ 6619.620213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6619.636108] Lustre: Skipped 1 previous similar message [ 6619.910350] Lustre: server umount lustre-MDT0000 complete [ 6623.437985] LustreError: 213632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421364 with bad export cookie 14281659890436175199 [ 6623.443194] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6623.449282] LustreError: 213632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6623.796995] Lustre: server umount lustre-MDT0001 complete [ 6634.532372] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6637.011271] Key type lgssc unregistered [ 6637.276536] LNet: 217338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6637.289561] LNetError: 217338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.309129] LNet: Removed LNI 192.168.206.118@tcp [ 6638.041393] Key type .llcrypt unregistered [ 6638.043471] Key type ._llcrypt unregistered [ 6657.424948] Key type ._llcrypt registered [ 6657.428076] Key type .llcrypt registered [ 6657.543686] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6658.844654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6658.915063] alg: No test for adler32 (adler32-zlib) [ 6660.106914] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6660.413956] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6662.128165] Key type lgssc registered [ 6663.588829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6672.278324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6672.305075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.750433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.878511] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6684.131448] LustreError: 218607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.154669] LustreError: 218607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6689.252400] LustreError: 218606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.610291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.216704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6697.107718] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6705.710115] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6711.916654] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.766446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.970294] LustreError: 220429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.358801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6724.594093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6725.640501] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6733.104237] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6742.415186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6742.519977] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6742.885269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6749.525611] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6757.696301] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6759.681790] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6771.516041] Lustre: Permanently deactivating lustre-OST0001 [ 6771.552890] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6780.493441] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6786.316831] Lustre: Permanently reactivating lustre-OST0001 [ 6786.331716] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6786.336772] Lustre: Skipped 2 previous similar messages [ 6787.588716] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.605409] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6787.615877] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6787.624715] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6787.656248] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6795.240930] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.251813] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6795.257333] Lustre: Skipped 2 previous similar messages [ 6797.339356] Lustre: server umount lustre-OST0001 complete [ 6800.355179] LustreError: 220431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.380977] LustreError: 220431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6801.377953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6801.390758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.416310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6801.421805] Lustre: Skipped 1 previous similar message [ 6802.402266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6805.483151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6805.488227] LustreError: 221104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.492942] Lustre: Skipped 2 previous similar messages [ 6805.517917] LustreError: 221104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6807.246431] Lustre: server umount lustre-OST0000 complete [ 6814.177783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.185902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.204210] Lustre: Skipped 1 previous similar message [ 6814.213622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.251584] Lustre: server umount lustre-MDT0000 complete [ 6820.734311] LustreError: 220456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421561 with bad export cookie 15716386423882318264 [ 6820.742347] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6820.744174] LustreError: 220456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6821.048886] Lustre: server umount lustre-MDT0001 complete [ 6831.368534] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6833.896656] Key type lgssc unregistered [ 6834.171045] LNet: 223660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6834.174952] LNetError: 223660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6834.189213] LNet: Removed LNI 192.168.206.118@tcp [ 6834.996374] Key type .llcrypt unregistered [ 6834.998133] Key type ._llcrypt unregistered [ 6872.688966] Key type ._llcrypt registered [ 6872.695505] Key type .llcrypt registered [ 6872.842569] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6873.883293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6873.922239] alg: No test for adler32 (adler32-zlib) [ 6874.934453] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6875.180567] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6876.832874] Key type lgssc registered [ 6878.077713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6885.460262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6885.477340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.834372] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6886.856544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6886.984169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.982181] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6898.640103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.700618] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6898.732887] Lustre: 225865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.921078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.202938] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6910.541735] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6916.462845] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6923.006766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6923.127585] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6923.326179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6929.964799] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6933.271646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6939.156228] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6943.314215] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6950.369618] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6957.006421] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6963.450606] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6975.224627] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6991.655108] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6991.662708] Lustre: Skipped 1 previous similar message [ 7026.921572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7026.933412] Lustre: Skipped 4 previous similar messages [ 7094.766837] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7094.787972] Lustre: Skipped 7 previous similar messages [ 7128.033401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7128.035878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.046280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7128.050880] Lustre: Skipped 1 previous similar message [ 7128.078325] Lustre: Skipped 1 previous similar message [ 7132.586580] Lustre: server umount lustre-OST0000 complete [ 7138.284363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.291195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7138.314303] Lustre: Skipped 1 previous similar message [ 7138.346517] Lustre: Skipped 1 previous similar message [ 7143.407872] Lustre: server umount lustre-MDT0000 complete [ 7147.315264] LustreError: 228437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421887 with bad export cookie 8402466313970957206 [ 7147.325206] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.329029] LustreError: 228437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7147.695342] Lustre: server umount lustre-MDT0001 complete [ 7160.341394] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7163.539688] Key type lgssc unregistered [ 7163.829850] LNet: 230711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7163.844437] LNetError: 230711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7163.879683] LNet: Removed LNI 192.168.206.118@tcp [ 7164.810316] Key type .llcrypt unregistered [ 7164.815262] Key type ._llcrypt unregistered [ 7187.328365] Key type ._llcrypt registered [ 7187.330109] Key type .llcrypt registered [ 7187.414335] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7188.710546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7188.790622] alg: No test for adler32 (adler32-zlib) [ 7189.953170] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 7190.310832] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7191.976166] Key type lgssc registered [ 7193.121607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7201.476059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7201.508613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.978520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.590454] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7212.521630] LustreError: 232125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.535892] LustreError: 232125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7215.867815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.282978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7220.137172] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7228.898604] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7234.581987] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7240.818967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7241.121602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.972653] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7248.308258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7254.639742] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7272.933040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.942415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.969952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.960347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.973341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.973540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.017476] Lustre: Skipped 1 previous similar message [ 7278.501252] Lustre: server umount lustre-OST0000 complete [ 7282.657203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7282.665224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.683300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.847166] Lustre: server umount lustre-MDT0000 complete [ 7289.317259] LustreError: 232125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.335705] LustreError: 232125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7292.534916] LustreError: 235063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422033 with bad export cookie 4483265748891737116 [ 7292.537953] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.552481] LustreError: 235063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7292.864285] Lustre: server umount lustre-MDT0001 complete [ 7303.279788] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7305.951201] Key type lgssc unregistered [ 7306.250052] LNet: 235818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7306.269651] LNetError: 235818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7306.295225] LNet: Removed LNI 192.168.206.118@tcp [ 7307.161620] Key type .llcrypt unregistered [ 7307.164509] Key type ._llcrypt unregistered [ 7328.891664] Key type ._llcrypt registered [ 7328.894700] Key type .llcrypt registered [ 7329.052499] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7330.438268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7330.585385] alg: No test for adler32 (adler32-zlib) [ 7331.686185] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 7332.025513] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7333.737222] Key type lgssc registered [ 7334.970299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7607.673104] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7610.251230] Key type lgssc unregistered [ 7610.491990] LNet: 237864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7610.498651] LNetError: 237864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7610.514643] LNet: Removed LNI 192.168.206.118@tcp [ 7611.384683] Key type .llcrypt unregistered [ 7611.389969] Key type ._llcrypt unregistered [ 7637.594975] Key type ._llcrypt registered [ 7637.597702] Key type .llcrypt registered [ 7637.728447] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7654.088455] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7655.259752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7655.281616] alg: No test for adler32 (adler32-zlib) [ 7656.330233] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 7656.649455] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7658.306273] Key type lgssc registered [ 7659.383537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7690.541743] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7345 sec ========== 00:27:09 (1786422429) [ 7692.737129] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:27:11 (1786422431) === [ 7696.850665] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:27:15 (1786422435) === [ 7738.011661] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7740.600379] Key type lgssc unregistered [ 7740.888854] LNet: 243104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7740.903685] LNetError: 243104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7740.921396] LNet: Removed LNI 192.168.206.118@tcp [ 7741.659469] Key type .llcrypt unregistered [ 7741.661332] Key type ._llcrypt unregistered