[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 636899337 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002430] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007709] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.014026] *** VALIDATE selinux *** [ 0.021792] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026379] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030075] *** VALIDATE tmpfs *** [ 0.031515] *** VALIDATE proc *** [ 0.032249] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034343] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036083] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038267] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042455] debug: unmapping init [mem 0xffffffff95459000-0xffffffff95460fff] [ 0.044460] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045669] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046040] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053246] rcu: Hierarchical SRCU implementation. [ 0.055567] smp: Bringing up secondary CPUs ... [ 0.056689] x86: Booting SMP configuration: [ 0.057038] .... node #0, CPUs: #1 #2 #3 [ 0.093715] smp: Brought up 1 node, 4 CPUs [ 0.095023] smpboot: Max logical packages: 1 [ 0.096018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142028] node 0 deferred pages initialised in 44ms [ 0.147282] devtmpfs: initialized [ 0.151072] x86/mm: Memory block size: 128MB [ 0.155684] gcov: version magic: 0x41383552 [ 0.159465] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165404] pinctrl core: initialized pinctrl subsystem [ 0.168189] [ 0.169007] ************************************************************* [ 0.173023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175009] ** ** [ 0.178012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181011] ** ** [ 0.184012] ** This means that this kernel is built to expose internal ** [ 0.186009] ** IOMMU data structures, which may compromise security on ** [ 0.189010] ** your system. ** [ 0.192012] ** ** [ 0.196014] ** If you see this message and you are not debugging the ** [ 0.199012] ** kernel, report this immediately to your vendor! ** [ 0.200009] ** ** [ 0.203012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.206012] ************************************************************* [ 0.208825] NET: Registered protocol family 16 [ 0.210917] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.214070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222217] cpuidle: using governor menu [ 0.225025] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228633] PCI: Using configuration type 1 for base access [ 0.231118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.240180] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.242349] cryptd: max_cpu_qlen set to 1000 [ 0.244272] ACPI: Added _OSI(Module Device) [ 0.246044] ACPI: Added _OSI(Processor Device) [ 0.248014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256600] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.267127] ACPI: Interpreter enabled [ 0.268059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.272014] ACPI: Using IOAPIC for interrupt routing [ 0.273000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.276565] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.298690] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.301040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.308021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322000] acpiphp: Slot [2] registered [ 0.328264] acpiphp: Slot [5] registered [ 0.333165] acpiphp: Slot [6] registered [ 0.336146] acpiphp: Slot [7] registered [ 0.343861] acpiphp: Slot [8] registered [ 0.349276] acpiphp: Slot [9] registered [ 0.353139] acpiphp: Slot [10] registered [ 0.357212] acpiphp: Slot [3] registered [ 0.361211] acpiphp: Slot [4] registered [ 0.364515] acpiphp: Slot [11] registered [ 0.367166] acpiphp: Slot [12] registered [ 0.370122] acpiphp: Slot [13] registered [ 0.371145] acpiphp: Slot [14] registered [ 0.373120] acpiphp: Slot [15] registered [ 0.375207] acpiphp: Slot [16] registered [ 0.377126] acpiphp: Slot [17] registered [ 0.384158] acpiphp: Slot [18] registered [ 0.385138] acpiphp: Slot [19] registered [ 0.389164] acpiphp: Slot [20] registered [ 0.393512] acpiphp: Slot [21] registered [ 0.395137] acpiphp: Slot [22] registered [ 0.398157] acpiphp: Slot [23] registered [ 0.405877] acpiphp: Slot [24] registered [ 0.412875] acpiphp: Slot [25] registered [ 0.420171] acpiphp: Slot [26] registered [ 0.421000] acpiphp: Slot [27] registered [ 0.421000] acpiphp: Slot [28] registered [ 0.436137] acpiphp: Slot [29] registered [ 0.448157] acpiphp: Slot [30] registered [ 0.460685] acpiphp: Slot [31] registered [ 0.461071] PCI host bridge to bus 0000:00 [ 0.465020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.469038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.482027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.487029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.490024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.496024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.503016] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.509000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.512869] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.533022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.541616] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.544015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.549025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.552014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.557623] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.572143] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.588047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.596101] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 23437 usecs [ 0.608093] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.626026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.655000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.661012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.667876] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.674014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.679011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.697015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.710140] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.724013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.735057] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.762014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.783729] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.798017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.818017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.866015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.895964] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.913017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.927018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.956020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.975154] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.991014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.004018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.044026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.065053] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.085019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.101013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.145014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.168025] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.172361] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.174311] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.182215] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.185194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.195045] iommu: Default domain type: Passthrough [ 1.196442] SCSI subsystem initialized [ 1.202160] ACPI: bus type USB registered [ 1.205163] usbcore: registered new interface driver usbfs [ 1.209092] usbcore: registered new interface driver hub [ 1.214193] usbcore: registered new device driver usb [ 1.216129] pps_core: LinuxPPS API ver. 1 registered [ 1.220014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.225061] PTP clock support registered [ 1.229196] EDAC MC: Ver: 3.0.0 [ 1.233116] PCI: Using ACPI for IRQ routing [ 1.234051] NetLabel: Initializing [ 1.236006] NetLabel: domain hash size = 128 [ 1.239010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.240085] NetLabel: unlabeled traffic allowed by default [ 1.243285] vgaarb: loaded [ 1.244694] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.247008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.254363] clocksource: Switched to clocksource kvm-clock [ 1.375314] VFS: Disk quotas dquot_6.6.0 [ 1.376332] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.378142] *** VALIDATE ramfs *** [ 1.378976] *** VALIDATE hugetlbfs *** [ 1.380065] pnp: PnP ACPI init [ 1.382025] pnp: PnP ACPI: found 6 devices [ 1.395971] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.399195] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.400700] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.402267] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.404144] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.406098] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.408200] NET: Registered protocol family 2 [ 1.409991] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.413751] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.416343] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.420478] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.423090] TCP: Hash tables configured (established 65536 bind 65536) [ 1.425292] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.427405] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.429453] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.431540] NET: Registered protocol family 1 [ 1.433996] RPC: Registered named UNIX socket transport module. [ 1.435368] RPC: Registered udp transport module. [ 1.436403] RPC: Registered tcp transport module. [ 1.437890] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.439596] NET: Registered protocol family 44 [ 1.440878] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.442466] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.443827] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.445467] PCI: CLS 0 bytes, default 64 [ 1.446613] Unpacking initramfs... [ 4.307434] debug: unmapping init [mem 0xffff9567bcc54000-0xffff9567bffbffff] [ 4.319533] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.324337] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 4.332109] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.287926] Initialise system trusted keyrings [ 5.305075] Key type blacklist registered [ 5.313576] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.360270] zbud: loaded [ 5.374407] *** VALIDATE nfs *** [ 5.377334] *** VALIDATE nfs4 *** [ 5.387470] pstore: using deflate compression [ 5.405532] Platform Keyring initialized [ 6.174198] NET: Registered protocol family 38 [ 6.176503] Key type asymmetric registered [ 6.177898] Asymmetric key parser 'x509' registered [ 6.180395] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.183346] io scheduler mq-deadline registered [ 6.184766] io scheduler kyber registered [ 6.186934] io scheduler bfq registered [ 6.188940] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.194114] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.197497] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.200367] ACPI: Power Button [PWRF] [ 6.207682] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.215910] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.244113] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.256539] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.284345] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.328994] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.362159] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.367766] Non-volatile memory driver v1.3 [ 6.369301] Linux agpgart interface v0.103 [ 6.400948] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 6.404258] vda: detected capacity change from 0 to 74838016 [ 6.460608] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.463596] vdb: detected capacity change from 0 to 1073741824 [ 6.481610] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.484982] vdc: detected capacity change from 0 to 2621440000 [ 6.523793] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.541085] vdd: detected capacity change from 0 to 2621440000 [ 6.576974] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.580480] vde: detected capacity change from 0 to 4294967296 [ 6.629279] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.633891] vdf: detected capacity change from 0 to 4294967296 [ 6.659428] libphy: Fixed MDIO Bus: probed [ 6.674988] usbcore: registered new interface driver usbserial_generic [ 6.682195] usbserial: USB Serial support registered for generic [ 6.686942] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.704532] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.707924] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.717009] mousedev: PS/2 mouse device common for all mice [ 6.726869] rtc_cmos 00:05: RTC can wake from S4 [ 6.733181] rtc_cmos 00:05: registered as rtc0 [ 6.741020] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.743072] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.766854] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.767652] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.799196] intel_pstate: CPU model not supported [ 6.805052] hid: raw HID events driver (C) Jiri Kosina [ 6.808467] usbcore: registered new interface driver usbhid [ 6.815104] usbhid: USB HID core driver [ 6.822362] drop_monitor: Initializing network drop monitor service [ 6.831126] Initializing XFRM netlink socket [ 6.836370] NET: Registered protocol family 10 [ 6.841121] Segment Routing with IPv6 [ 6.843121] NET: Registered protocol family 17 [ 6.846632] mpls_gso: MPLS GSO support [ 6.854708] RAS: Correctable Errors collector initialized. [ 6.864115] AVX version of gcm_enc/dec engaged. [ 6.871590] AES CTR mode by8 optimization enabled [ 7.003547] sched_clock: Marking stable (7003517390, 0)->(8280527593, -1277010203) [ 7.008236] registered taskstats version 1 [ 7.010933] Loading compiled-in X.509 certificates [ 7.013921] zswap: loaded using pool lzo/zbud [ 7.043937] Key type big_key registered [ 7.064709] Key type encrypted registered [ 7.066662] ima: No TPM chip found, activating TPM-bypass! [ 7.069354] ima: Allocated hash algorithm: sha1 [ 7.071644] ima: No architecture policies found [ 7.074729] evm: Initialising EVM extended attributes: [ 7.077906] evm: security.selinux [ 7.079518] evm: security.ima [ 7.081179] evm: security.capability [ 7.082987] evm: HMAC attrs: 0x1 [ 7.087496] rtc_cmos 00:05: setting system clock to 2026-09-01 13:26:48 UTC (1788269208) [ 7.096950] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 7.103986] debug: unmapping init [mem 0xffffffff95182000-0xffffffff95458fff] [ 7.115253] Write protecting the kernel read-only data: 28672k [ 7.122479] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 7.127169] debug: unmapping init [mem 0xffffffff94114000-0xffffffff941fffff] [ 7.189316] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.231565] systemd[1]: Detected virtualization kvm. [ 7.243482] systemd[1]: Detected architecture x86-64. [ 7.247158] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.314902] systemd[1]: No hostname configured. [ 7.320132] systemd[1]: Set hostname to . [ 7.325762] random: systemd: uninitialized urandom read (16 bytes read) [ 7.334147] systemd[1]: Initializing machine ID from random generator. [ 7.467301] random: ln: uninitialized urandom read (6 bytes read) [ 7.763946] random: systemd: uninitialized urandom read (16 bytes read) [ 7.773174] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.806321] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.815141] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.781908] device-mapper: uevent: version 1.0.3 [ 9.784490] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.547972] random: fast init done [ 12.636813] virtio_net virtio0 ens2: renamed from eth0 [ 12.876250] scsi host0: ata_piix [ 12.920240] scsi host1: ata_piix [ 12.921691] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.926732] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.607347] random: crng init done [ 17.608940] random: 7 urandom warning(s) missed due to ratelimiting [ 17.924182] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.116876] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.280587] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.187877] SELinux: Disabled at runtime. [ 25.343286] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.368772] systemd[1]: Detected virtualization kvm. [ 25.372548] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.462215] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.476604] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.503356] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.518530] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.537721] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.564642] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.588898] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ 28.009485] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 29.304062] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.563027] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.598823] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.367880] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.491950] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.135572] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.660046] NFS: Registering the id_resolver key type [ 36.666233] Key type id_resolver registered [ 36.667673] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg338-server login: [ 75.863914] libcfs: loading out-of-tree module taints kernel. [ 75.885678] Key type ._llcrypt registered [ 75.887197] Key type .llcrypt registered [ 75.944266] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 86.364464] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 87.079498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.089925] alg: No test for adler32 (adler32-zlib) [ 88.194887] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 88.597915] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 90.271164] Key type lgssc registered [ 91.131119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.492321] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.919971] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 130.543462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 130.562489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.677268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.701851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.762568] Lustre: lustre-MDT0000: new disk, initializing [ 131.811173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.836778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.839763] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.721634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.769551] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 141.799368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 141.802786] Lustre: Skipped 1 previous similar message [ 141.862275] Lustre: lustre-MDT0001: new disk, initializing [ 141.907120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 141.927960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 141.942090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 144.187634] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.205879] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 152.419652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 152.564388] Lustre: lustre-OST0000: new disk, initializing [ 152.567195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 152.571234] Lustre: 8422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 152.598871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.187886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 154.194304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 154.255119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 156.182970] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.697958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 165.773393] Lustre: lustre-OST0001: new disk, initializing [ 165.779800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 165.784803] Lustre: 9492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 165.835150] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.284843] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.545986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 172.554373] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 172.579142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 177.017492] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.726889] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 184.040673] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing check_logdir /tmp/testlogs/ [ 186.821696] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing yml_node [ 189.372455] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 190.850576] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 192.405983] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 193.444673] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 1 09:29:53 EDT 2026 [ 202.958889] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 203.856313] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 218.591704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.593104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 218.597703] Lustre: Skipped 3 previous similar messages [ 218.598050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 218.611964] Lustre: Skipped 3 previous similar messages [ 222.892142] Lustre: server umount lustre-MDT0000 complete [ 223.711540] LustreError: 8357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 223.723240] LustreError: 8357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 224.715769] LustreError: 8421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269426 with bad export cookie 2100640189463804223 [ 224.717964] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 224.721606] LustreError: 8421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 228.831971] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 228.832359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 228.852412] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 228.876338] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 228.884981] Lustre: Skipped 1 previous similar message [ 229.855678] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 229.862967] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 234.976414] LustreError: 9001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 234.976470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 234.992243] LustreError: 9001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 234.997934] Lustre: Skipped 3 previous similar messages [ 239.071264] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 239.207890] Lustre: server umount lustre-MDT0001 complete [ 241.880403] Lustre: server umount lustre-OST0000 complete [ 244.528822] Lustre: server umount lustre-OST0001 complete [ 248.913094] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 255.040551] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 276.899697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.075675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 277.111788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 277.162277] Lustre: lustre-MDT0000: new disk, initializing [ 277.207874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 277.221463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 279.554608] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 285.778239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 285.846464] Lustre: 15043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 285.884300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 285.887132] Lustre: Skipped 1 previous similar message [ 285.930143] Lustre: lustre-MDT0001: new disk, initializing [ 285.990924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 286.012265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 286.022064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 288.193518] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 293.468766] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 296.165448] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 299.557526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.737593] Lustre: 17010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 301.219148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 301.317548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 303.090855] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 309.237387] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 311.801174] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 311.951639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 314.698887] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 314.845068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 316.905019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.906647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 316.906980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 316.919717] Lustre: Skipped 2 previous similar messages [ 322.019577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 322.027504] Lustre: Skipped 2 previous similar messages [ 322.180269] Lustre: server umount lustre-OST0000 complete [ 327.142944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.144929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 327.160628] LustreError: Skipped 1 previous similar message [ 330.890634] Lustre: server umount lustre-MDT0000 complete [ 332.257331] LustreError: 16049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.269024] LustreError: 16049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 333.084839] LustreError: 15837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269534 with bad export cookie 2100640189463805749 [ 333.091913] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.279018] Lustre: server umount lustre-MDT0001 complete [ 335.425190] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:32:15 (1788269535) [ 339.557616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.865317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.869923] Lustre: Skipped 1 previous similar message [ 342.209258] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 345.061285] LustreError: 19276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.072431] LustreError: 19276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 347.082077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.138469] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 355.221687] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 359.413940] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 363.677897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.920760] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 372.119180] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 377.315216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 377.316021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.327776] Lustre: Skipped 2 previous similar messages [ 377.328411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 377.336763] Lustre: Skipped 3 previous similar messages [ 379.044897] Lustre: server umount lustre-OST0000 complete [ 387.462238] Lustre: server umount lustre-MDT0000 complete [ 387.552317] LustreError: 21115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.569216] LustreError: 21115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 389.797372] LustreError: 21116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269591 with bad export cookie 2100640189463806638 [ 389.798641] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.808836] LustreError: 21116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 390.061912] Lustre: server umount lustre-MDT0001 complete [ 396.734417] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 398.655281] Key type lgssc unregistered [ 398.904142] LNet: 22807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.913397] LNetError: 22807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 398.934952] LNet: Removed LNI 192.168.203.138@tcp [ 399.557290] Key type .llcrypt unregistered [ 399.564330] Key type ._llcrypt unregistered [ 414.657122] Key type ._llcrypt registered [ 414.659682] Key type .llcrypt registered [ 414.735721] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 415.340852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 415.361185] alg: No test for adler32 (adler32-zlib) [ 416.331852] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 416.493428] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 418.119267] Key type lgssc registered [ 418.930414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 424.059994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 424.071667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.337568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.077718] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 433.502743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.735330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.090329] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 440.585537] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 444.267313] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 448.112242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.341464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.412270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 451.365963] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 455.686545] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.890322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.902673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.918857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.962535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.966093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.966341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.979202] Lustre: Skipped 1 previous similar message [ 470.960965] Lustre: server umount lustre-OST0000 complete [ 474.083792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.099887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.104851] Lustre: Skipped 1 previous similar message [ 479.202862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 479.209868] Lustre: Skipped 1 previous similar message [ 479.433449] Lustre: server umount lustre-MDT0000 complete [ 481.647069] LustreError: 24176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269683 with bad export cookie 2891092199815634477 [ 481.648369] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 481.655840] LustreError: 24176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 481.849940] Lustre: server umount lustre-MDT0001 complete [ 488.260867] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 490.084626] Key type lgssc unregistered [ 490.314688] LNet: 28171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.324978] LNetError: 28171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 490.339334] LNet: Removed LNI 192.168.203.138@tcp [ 490.879485] Key type .llcrypt unregistered [ 490.881065] Key type ._llcrypt unregistered [ 503.956314] Key type ._llcrypt registered [ 503.958123] Key type .llcrypt registered [ 504.022739] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 504.456848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 504.484090] alg: No test for adler32 (adler32-zlib) [ 505.427380] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 505.570201] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 507.175169] Key type lgssc registered [ 507.780773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 512.291245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 512.302635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.580194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.817412] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 520.243325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.450078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 522.637587] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 526.228984] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.815243] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.149704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.345269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.419049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 540.103636] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 544.037637] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 548.833442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.841658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.851657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.880929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.883752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.889490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.230532] Lustre: server umount lustre-OST0000 complete [ 556.516119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.524910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.532432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.534404] Lustre: Skipped 1 previous similar message [ 559.015723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.018331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.519313] Lustre: server umount lustre-MDT0000 complete [ 564.681032] LustreError: 29542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269766 with bad export cookie 8952258032664553365 [ 564.684240] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.687885] LustreError: 29542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 564.842125] Lustre: server umount lustre-MDT0001 complete [ 570.341114] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 571.815217] Key type lgssc unregistered [ 572.013109] LNet: 33436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.017820] LNetError: 33436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 572.030628] LNet: Removed LNI 192.168.203.138@tcp [ 572.520158] Key type .llcrypt unregistered [ 572.523043] Key type ._llcrypt unregistered [ 585.295636] Key type ._llcrypt registered [ 585.297337] Key type .llcrypt registered [ 585.352653] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 585.816099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 585.887489] alg: No test for adler32 (adler32-zlib) [ 586.813399] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 586.947571] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 588.575318] Key type lgssc registered [ 589.324614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 594.120523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 594.138400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.434845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.967835] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 603.549806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.757210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 606.000385] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 609.836414] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 612.793558] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.710630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 616.908655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.009837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 620.541143] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 624.895933] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 635.365655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.372217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.383026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 639.456457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 639.459541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.460831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 639.475161] Lustre: Skipped 1 previous similar message [ 644.583059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 644.587866] Lustre: Skipped 1 previous similar message [ 649.183104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 649.283642] Lustre: server umount lustre-OST0000 complete [ 654.817629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.818477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.828312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 654.838100] Lustre: Skipped 1 previous similar message [ 657.595255] Lustre: server umount lustre-MDT0000 complete [ 659.633574] LustreError: 34806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269861 with bad export cookie 2268834743036798541 [ 659.635525] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.639904] LustreError: 34806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.801528] Lustre: server umount lustre-MDT0001 complete [ 665.488958] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 667.077590] Key type lgssc unregistered [ 667.256612] LNet: 38360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.260794] LNetError: 38360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.274655] LNet: Removed LNI 192.168.203.138@tcp [ 667.761391] Key type .llcrypt unregistered [ 667.763170] Key type ._llcrypt unregistered [ 680.710471] Key type ._llcrypt registered [ 680.712063] Key type .llcrypt registered [ 680.773541] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 681.353672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 681.403123] alg: No test for adler32 (adler32-zlib) [ 682.345474] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 682.525297] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 684.167200] Key type lgssc registered [ 684.973940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 689.846427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.857191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.190758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.477926] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 698.346736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.565059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 700.826800] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 704.854746] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 707.989876] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 712.063423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.270520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.959022] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 716.401890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 720.184871] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 729.061108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.065124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 729.068459] Lustre: Skipped 1 previous similar message [ 729.074503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.935640] Lustre: server umount lustre-OST0000 complete [ 749.025500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.032143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.033083] Lustre: Skipped 1 previous similar message [ 749.045217] Lustre: Skipped 2 previous similar messages [ 754.145858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.150368] Lustre: Skipped 1 previous similar message [ 759.264598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.269313] Lustre: Skipped 1 previous similar message [ 760.287123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 760.367774] Lustre: server umount lustre-MDT0000 complete [ 762.394883] LustreError: 39733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269963 with bad export cookie 2344396160021842802 [ 762.398226] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 762.401159] LustreError: 39733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 768.587738] Lustre: server umount lustre-MDT0001 complete [ 773.947580] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 775.527877] Key type lgssc unregistered [ 775.725382] LNet: 43457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.735225] LNetError: 43457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.743745] LNet: Removed LNI 192.168.203.138@tcp [ 776.271046] Key type .llcrypt unregistered [ 776.273070] Key type ._llcrypt unregistered [ 788.931595] Key type ._llcrypt registered [ 788.933924] Key type .llcrypt registered [ 789.001781] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 789.543808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.576677] alg: No test for adler32 (adler32-zlib) [ 790.499752] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 790.641146] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 792.247338] Key type lgssc registered [ 792.877783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 797.233247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 797.244074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.465101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.633402] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 804.956888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.139742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 807.152468] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 811.081198] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 813.945363] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 817.666520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 817.873385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.973540] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 823.018141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 824.603957] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 835.553862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.558072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.573546] Lustre: Skipped 1 previous similar message [ 837.189148] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 837.192470] Lustre: Skipped 2 previous similar messages [ 839.375921] Lustre: server umount lustre-MDT0000 complete [ 840.675174] LustreError: 44847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.684194] LustreError: 44847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 841.542736] LustreError: 44829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270042 with bad export cookie 17920046483619976018 [ 841.547308] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.553121] LustreError: 44829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 841.695324] LustreError: 45576:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000debba8e9 x1875136995155712/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 842.289375] LustreError: 44843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.302270] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.38@tcp (stopping) [ 845.793199] LustreError: 44844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.793839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.811113] Lustre: Skipped 1 previous similar message [ 845.814781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 847.795120] Lustre: server umount lustre-MDT0001 complete [ 866.353412] Lustre: server umount lustre-OST0000 complete [ 873.643344] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 875.025161] Key type lgssc unregistered [ 875.175464] LNet: 48704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.179515] LNetError: 48704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.189918] LNet: Removed LNI 192.168.203.138@tcp [ 875.579329] Key type .llcrypt unregistered [ 875.580757] Key type ._llcrypt unregistered [ 887.443225] Key type ._llcrypt registered [ 887.446502] Key type .llcrypt registered [ 887.496843] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 887.900863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 887.937894] alg: No test for adler32 (adler32-zlib) [ 888.859692] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 889.015791] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 890.631164] Key type lgssc registered [ 891.117256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.474952] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.484976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.919393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 915.039315] LustreError: 50067:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 941.663285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.000349] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 946.864063] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1111.047790] Lustre: server umount lustre-OST0000 complete [ 1126.039956] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1128.882312] Key type lgssc unregistered [ 1129.084806] LNet: 51754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.089946] LNetError: 51754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.101387] LNet: Removed LNI 192.168.203.138@tcp [ 1129.677758] Key type .llcrypt unregistered [ 1129.690374] Key type ._llcrypt unregistered [ 1155.047491] Key type ._llcrypt registered [ 1155.057851] Key type .llcrypt registered [ 1155.247665] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1156.969715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.048585] alg: No test for adler32 (adler32-zlib) [ 1158.554558] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1158.865704] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1158.971180] hrtimer: interrupt took 5170485 ns [ 1160.679200] Key type lgssc registered [ 1162.384427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1173.226108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1173.249373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.962801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.137653] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1185.250299] LustreError: 53169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.268774] LustreError: 53169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1187.416343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.751177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.730124] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1196.378887] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1200.228537] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1204.741579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.165941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.764514] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1213.688186] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1218.532498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.534810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1218.541025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.576714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.580894] Lustre: Skipped 1 previous similar message [ 1221.181971] Lustre: server umount lustre-OST0000 complete [ 1223.651038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1223.660049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.671229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.699526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.700210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.714021] Lustre: Skipped 1 previous similar message [ 1229.582947] Lustre: server umount lustre-MDT0000 complete [ 1230.819032] LustreError: 54858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.841967] LustreError: 54858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1232.115938] LustreError: 55949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270433 with bad export cookie 15988705579714133760 [ 1232.121128] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.123720] LustreError: 55949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1232.376299] Lustre: server umount lustre-MDT0001 complete [ 1239.548759] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1241.418486] Key type lgssc unregistered [ 1241.603776] LNet: 56702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.610559] LNetError: 56702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.626407] LNet: Removed LNI 192.168.203.138@tcp [ 1242.205208] Key type .llcrypt unregistered [ 1242.207383] Key type ._llcrypt unregistered [ 1256.990779] Key type ._llcrypt registered [ 1256.992878] Key type .llcrypt registered [ 1257.066977] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1257.823384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.837138] alg: No test for adler32 (adler32-zlib) [ 1258.783616] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1258.992519] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1260.615310] Key type lgssc registered [ 1261.374307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.004955] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.025794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.504122] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1286.623312] LustreError: 58077:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 1313.311497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.045254] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1323.181733] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1327.843949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.214168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.275660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1331.315350] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1334.754804] LustreError: 58102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.773265] LustreError: 58102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1337.585088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.873655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1340.988639] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1346.087726] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1350.035420] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1353.195438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.208185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.208262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.714761] Lustre: server umount lustre-OST0000 complete [ 1377.770226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.777550] Lustre: Skipped 1 previous similar message [ 1377.780888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.789352] Lustre: Skipped 2 previous similar messages [ 1382.884173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.891530] Lustre: Skipped 1 previous similar message [ 1383.738792] Lustre: server umount lustre-MDT0000 complete [ 1386.559373] LustreError: 58084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270587 with bad export cookie 9853982726284093105 [ 1386.566864] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.576059] LustreError: 58084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1392.906600] Lustre: server umount lustre-MDT0001 complete [ 1400.743862] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1402.787926] Key type lgssc unregistered [ 1403.036209] LNet: 61822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.045468] LNetError: 61822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.060739] LNet: Removed LNI 192.168.203.138@tcp [ 1403.885712] Key type .llcrypt unregistered [ 1403.891317] Key type ._llcrypt unregistered [ 1424.323948] Key type ._llcrypt registered [ 1424.326934] Key type .llcrypt registered [ 1424.419508] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1425.274119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.290293] alg: No test for adler32 (adler32-zlib) [ 1426.534796] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1426.880847] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1428.567175] Key type lgssc registered [ 1429.673311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.654946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.689371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.269980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.062583] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1449.450381] LustreError: 63217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.467681] LustreError: 63217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1450.661178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.988413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.005591] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1459.524619] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.600505] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.478315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1468.750701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.352861] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1479.312575] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1484.150168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1496.546088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.553750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.566143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.618851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.619356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.620263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1500.958988] Lustre: server umount lustre-OST0000 complete [ 1504.743927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.749638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.766621] Lustre: Skipped 1 previous similar message [ 1504.769789] Lustre: Skipped 1 previous similar message [ 1509.826067] Lustre: server umount lustre-MDT0000 complete [ 1509.865954] LustreError: 63218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.879441] LustreError: 63218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1512.663437] LustreError: 66020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270714 with bad export cookie 8264748131567656521 [ 1512.668632] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.675029] LustreError: 66020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1512.936804] Lustre: server umount lustre-MDT0001 complete [ 1520.746908] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1523.117408] Key type lgssc unregistered [ 1523.422303] LNet: 66775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.429911] LNetError: 66775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.446636] LNet: Removed LNI 192.168.203.138@tcp [ 1524.063972] Key type .llcrypt unregistered [ 1524.066479] Key type ._llcrypt unregistered [ 1542.380592] Key type ._llcrypt registered [ 1542.383709] Key type .llcrypt registered [ 1542.453700] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1543.102893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.191544] alg: No test for adler32 (adler32-zlib) [ 1544.223987] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1544.433940] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1546.079368] Key type lgssc registered [ 1547.170763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1553.367779] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:52:33 (1788270753) [ 1557.588783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.607687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.916841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.418246] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1569.252077] LustreError: 68456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.266714] LustreError: 68456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1569.377774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.662676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1573.375224] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1578.632337] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1582.556471] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1587.636480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.876711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1592.450122] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1597.597245] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1604.279394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1611.735166] Lustre: Failing over lustre-MDT0000 [ 1611.947392] Lustre: server umount lustre-MDT0000 complete [ 1613.405795] LustreError: 68451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.817675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.821459] LustreError: 70703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.833842] Lustre: Skipped 2 previous similar messages [ 1614.854914] LustreError: 70703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1618.486294] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.308664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.408104] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.502257] Lustre: *** cfs_fail_loc=135, val=0*** [ 1619.558400] LustreError: 71563:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1619.562218] LustreError: 71563:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -2 [ 1619.568639] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1619.568639] [ 1619.575366] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1619.589260] LustreError: 71551:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1619.594099] LustreError: 71551:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 1619.601967] LustreError: 71551:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1619.607329] LustreError: 71551:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1619.629167] Lustre: server umount lustre-MDT0000 complete [ 1619.635582] LustreError: 71551:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1623.597168] LustreError: 70703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.615412] LustreError: 70703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1624.706329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.800309] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1624.955054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.997119] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1627.864207] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1628.729610] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1630.195965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1630.224136] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1630.273649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1635.297749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.306437] Lustre: Skipped 1 previous similar message [ 1635.306506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.356380] Lustre: server umount lustre-OST0000 complete [ 1645.537780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1645.540503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.542707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.549818] Lustre: Skipped 1 previous similar message [ 1645.553193] Lustre: Skipped 1 previous similar message [ 1650.658735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.666035] Lustre: Skipped 2 previous similar messages [ 1654.757200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.763596] Lustre: Skipped 1 previous similar message [ 1656.287678] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1656.374451] Lustre: server umount lustre-MDT0000 complete [ 1658.321062] LustreError: 71055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270859 with bad export cookie 5364461232896655458 [ 1658.325243] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1658.327539] LustreError: 71055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1658.480946] Lustre: server umount lustre-MDT0001 complete [ 1663.763740] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1665.339490] Key type lgssc unregistered [ 1665.521964] LNet: 73344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.529712] LNetError: 73344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.550120] LNet: Removed LNI 192.168.203.138@tcp [ 1666.058921] Key type .llcrypt unregistered [ 1666.061046] Key type ._llcrypt unregistered [ 1679.661805] Key type ._llcrypt registered [ 1679.663741] Key type .llcrypt registered [ 1679.718733] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1680.306931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.347335] alg: No test for adler32 (adler32-zlib) [ 1681.298047] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1681.476753] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1683.103164] Key type lgssc registered [ 1683.822170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1688.397215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1688.412556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.663926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.886061] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1696.120959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.328598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.416361] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1702.114691] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1704.787938] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1707.937577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.124045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.701299] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1714.448293] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1715.308868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1728.132330] Lustre: Failing over lustre-MDT0000 [ 1728.301389] Lustre: server umount lustre-MDT0000 complete [ 1732.065468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.072125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.083625] LustreError: 74734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.092742] LustreError: 74734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1732.965035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.050138] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.090439] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.091275] LustreError: 74733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.116069] LustreError: 74733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1733.152918] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1733.214644] LustreError: 77837:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1733.217940] LustreError: 77837:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -9 [ 1733.221479] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1733.221479] [ 1733.226110] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1733.234874] LustreError: 77826:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1733.240400] LustreError: 77826:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 1733.245071] LustreError: 77826:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1733.249844] LustreError: 77826:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1733.266513] Lustre: server umount lustre-MDT0000 complete [ 1733.269079] LustreError: 77826:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1736.575566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.655072] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.806107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.838861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1737.262894] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1738.877950] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1742.312472] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1742.328338] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1742.351395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1747.431751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.431799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.438587] Lustre: Skipped 1 previous similar message [ 1747.445771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.448671] Lustre: Skipped 1 previous similar message [ 1750.309282] Lustre: server umount lustre-OST0000 complete [ 1752.545826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.547247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.547880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.557476] Lustre: Skipped 2 previous similar messages [ 1757.671254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.673861] Lustre: Skipped 1 previous similar message [ 1762.784274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.787989] Lustre: Skipped 1 previous similar message [ 1766.367128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.438621] Lustre: server umount lustre-MDT0000 complete [ 1766.880530] LustreError: 74734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.890063] LustreError: 74734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1768.068496] LustreError: 74715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270969 with bad export cookie 14701662772421129301 [ 1768.073130] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.073680] LustreError: 74715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1768.230947] Lustre: server umount lustre-MDT0001 complete [ 1772.682881] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1773.883651] Key type lgssc unregistered [ 1774.025458] LNet: 79623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.029674] LNetError: 79623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.037452] LNet: Removed LNI 192.168.203.138@tcp [ 1774.381846] Key type .llcrypt unregistered [ 1774.385076] Key type ._llcrypt unregistered [ 1779.546423] Key type ._llcrypt registered [ 1779.548105] Key type .llcrypt registered [ 1779.593774] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1779.742299] Key type .llcrypt unregistered [ 1779.744335] Key type ._llcrypt unregistered [ 1790.132773] Key type ._llcrypt registered [ 1790.134165] Key type .llcrypt registered [ 1790.180652] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1790.662830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.680814] alg: No test for adler32 (adler32-zlib) [ 1791.548093] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1791.650362] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1793.239151] Key type lgssc registered [ 1793.696429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.754790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.767350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.940118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.454732] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1802.725700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.867120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.329376] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1806.875949] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1809.095883] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1811.807913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.954274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.286866] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1817.277479] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1823.210172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1833.442967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.444146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.452618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.454840] Lustre: Skipped 1 previous similar message [ 1837.009295] Lustre: server umount lustre-OST0000 complete [ 1838.560654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.560991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.562970] LustreError: Skipped 1 previous similar message [ 1838.566571] Lustre: Skipped 1 previous similar message [ 1838.568449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.571545] Lustre: Skipped 1 previous similar message [ 1843.680729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.684118] Lustre: Skipped 1 previous similar message [ 1844.438589] Lustre: server umount lustre-MDT0000 complete [ 1845.792874] LustreError: 81500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271047 with bad export cookie 17429279661900148787 [ 1845.796975] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.798172] LustreError: 81500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1845.930688] Lustre: server umount lustre-MDT0001 complete [ 1849.837420] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1850.935115] Key type lgssc unregistered [ 1851.067510] LNet: 85072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.071039] LNetError: 85072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.078298] LNet: Removed LNI 192.168.203.138@tcp [ 1851.410643] Key type .llcrypt unregistered [ 1851.412440] Key type ._llcrypt unregistered [ 1860.855681] Key type ._llcrypt registered [ 1860.857772] Key type .llcrypt registered [ 1860.898673] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1861.371625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1861.382094] alg: No test for adler32 (adler32-zlib) [ 1862.251221] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1862.355426] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1863.943218] Key type lgssc registered [ 1864.335151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1867.221634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1867.231545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.404547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.767199] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1872.652195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.786245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.102598] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1876.435385] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1878.280583] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.524848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.642104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.580604] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1884.966517] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.969960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1898.466523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.467070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1898.467608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1898.470957] Lustre: Skipped 1 previous similar message [ 1899.859321] Lustre: server umount lustre-OST0000 complete [ 1903.252832] Lustre: server umount lustre-MDT0000 complete [ 1903.584904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.585641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.586470] LustreError: 86461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.586476] LustreError: 86461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1903.588420] LustreError: Skipped 1 previous similar message [ 1903.601188] Lustre: Skipped 1 previous similar message [ 1904.542075] LustreError: 86442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271105 with bad export cookie 15305672148468437674 [ 1904.545705] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.546941] LustreError: 86442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.667971] Lustre: server umount lustre-MDT0001 complete [ 1908.464620] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1909.593562] Key type lgssc unregistered [ 1909.723425] LNet: 90013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.726247] LNetError: 90013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.738282] LNet: Removed LNI 192.168.203.138@tcp [ 1910.047586] Key type .llcrypt unregistered [ 1910.048787] Key type ._llcrypt unregistered [ 1919.218351] Key type ._llcrypt registered [ 1919.219511] Key type .llcrypt registered [ 1919.264489] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1919.759912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1919.773699] alg: No test for adler32 (adler32-zlib) [ 1920.640233] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1920.726787] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1922.311163] Key type lgssc registered [ 1922.747861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.581554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.589353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.743475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.107047] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1930.980710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.127295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.627084] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1935.096741] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1936.847583] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1939.108118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.230993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.147310] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1943.544805] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1952.551638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1961.952730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1961.955865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.960297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1962.978193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.978447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.978561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1963.981308] Lustre: server umount lustre-OST0000 complete [ 1967.071819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1967.074663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.079306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.081077] Lustre: Skipped 1 previous similar message [ 1971.422322] Lustre: server umount lustre-MDT0000 complete [ 1972.845825] LustreError: 93899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271174 with bad export cookie 4308171300146796376 [ 1972.849722] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.849853] LustreError: 93899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1972.972752] Lustre: server umount lustre-MDT0001 complete [ 1977.067478] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1978.255462] Key type lgssc unregistered [ 1978.388079] LNet: 94956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.391702] LNetError: 94956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.400414] LNet: Removed LNI 192.168.203.138@tcp [ 1978.758212] Key type .llcrypt unregistered [ 1978.759872] Key type ._llcrypt unregistered [ 1988.546060] Key type ._llcrypt registered [ 1988.547597] Key type .llcrypt registered [ 1988.589987] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1988.989369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1989.035913] alg: No test for adler32 (adler32-zlib) [ 1989.920105] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1990.026903] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1991.615128] Key type lgssc registered [ 1992.054669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1994.929631] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1994.939654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2010.399413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2015.519228] LustreError: 96319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2042.143227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.028322] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2046.453895] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2048.439261] Lustre: server umount lustre-OST0000 complete [ 2051.276887] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:00:52 (1788271252) [ 2053.746553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.916421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.370738] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2058.429852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.558057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2060.103248] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2062.777290] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2064.720250] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.168896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2067.290453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.337737] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2071.791275] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2079.590127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2088.927870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2088.930917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.936249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.953188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.953663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.958742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.728133] Lustre: server umount lustre-OST0000 complete [ 2094.559843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2094.562512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.567567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.569339] Lustre: Skipped 1 previous similar message [ 2100.191916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.194077] Lustre: Skipped 3 previous similar messages [ 2105.312280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.314176] Lustre: Skipped 1 previous similar message [ 2107.359147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2107.421428] Lustre: server umount lustre-MDT0000 complete [ 2108.652285] LustreError: 97935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271310 with bad export cookie 4506057205292147340 [ 2108.655384] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2108.657642] LustreError: 97935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2108.764473] Lustre: server umount lustre-MDT0001 complete [ 2112.282948] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2113.321732] Key type lgssc unregistered [ 2113.449469] LNet: 101682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.451681] LNetError: 101682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.459384] LNet: Removed LNI 192.168.203.138@tcp [ 2113.759996] Key type .llcrypt unregistered [ 2113.761368] Key type ._llcrypt unregistered [ 2128.252130] Key type ._llcrypt registered [ 2128.253666] Key type .llcrypt registered [ 2128.290140] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2128.608769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2128.637980] alg: No test for adler32 (adler32-zlib) [ 2129.487356] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2129.578469] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2131.159198] Key type lgssc registered [ 2131.544548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2134.230474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2134.237299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.374949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.758485] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2139.653824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.781937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.072403] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2143.434280] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2145.197341] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2147.272749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2147.370413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.114190] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2151.212158] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2159.658596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2164.704789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.705620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.707030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.715626] Lustre: Skipped 1 previous similar message [ 2166.290267] Lustre: server umount lustre-OST0000 complete [ 2169.622281] Lustre: server umount lustre-MDT0000 complete [ 2169.825471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.831297] LustreError: 103843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.836964] LustreError: 103843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2170.868089] LustreError: 103825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271372 with bad export cookie 3246957159511519318 [ 2170.871695] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.873771] LustreError: 103825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2170.974630] Lustre: server umount lustre-MDT0001 complete [ 2174.430764] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2175.490298] Key type lgssc unregistered [ 2175.621374] LNet: 107397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.623940] LNetError: 107397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.636323] LNet: Removed LNI 192.168.203.138@tcp [ 2175.938198] Key type .llcrypt unregistered [ 2175.939260] Key type ._llcrypt unregistered [ 2183.054746] Key type ._llcrypt registered [ 2183.055879] Key type .llcrypt registered [ 2183.101647] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2183.646983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.670586] alg: No test for adler32 (adler32-zlib) [ 2184.548185] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2184.652524] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2186.239187] Key type lgssc registered [ 2186.637708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2189.320231] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2189.326603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2204.767476] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2209.887196] LustreError: 108617:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2236.512260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.324269] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2251.065196] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2257.285615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.411544] LustreError: 107954:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000132b9695 x1875138456848640/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2257.673222] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2257.678922] LustreError: 109640:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2257.682104] LustreError: 109640:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2257.688297] LustreError: 109640:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2257.748823] Lustre: server umount lustre-MDT0000 complete [ 2257.751590] LustreError: 109640:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2278.879438] Lustre: 108629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788271464/real 1788271464] req@00000000a89cda82 x1875138456849920/t0(0) o101->MGC192.168.203.138@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788271480 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2278.897171] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2278.906722] LustreError: 108629:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2278.988596] Lustre: server umount lustre-OST0000 complete [ 2283.666221] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2288.174662] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2304.867902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.966311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2304.983149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2305.022095] Lustre: lustre-MDT0000: new disk, initializing [ 2305.055981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.063319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2306.888219] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2312.456376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.534652] Lustre: 112427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2312.588789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2312.592957] Lustre: Skipped 1 previous similar message [ 2312.654312] Lustre: lustre-MDT0001: new disk, initializing [ 2312.755134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.792021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2312.803121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2315.194647] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2323.777640] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2330.356958] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.436541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.838770] Lustre: lustre-OST0000: new disk, initializing [ 2339.842431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2339.847802] Lustre: 114394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2339.950700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.581577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2346.597543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2346.689398] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2346.775129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2357.346106] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2362.827187] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2363.082747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2369.715615] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2370.002888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2372.576306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.580826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.588796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.597288] Lustre: Skipped 1 previous similar message [ 2377.706982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.722695] Lustre: Skipped 1 previous similar message [ 2378.771977] Lustre: server umount lustre-OST0000 complete [ 2382.817809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.826513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.826945] Lustre: Skipped 2 previous similar messages [ 2382.835731] Lustre: Skipped 1 previous similar message [ 2387.936260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.944828] Lustre: Skipped 1 previous similar message [ 2388.171645] Lustre: server umount lustre-MDT0000 complete [ 2391.963756] LustreError: 113221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271593 with bad export cookie 16748455848933677503 [ 2391.969350] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.979115] LustreError: 113221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2392.225186] Lustre: server umount lustre-MDT0001 complete [ 2401.069627] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:06:40 (1788271600) [ 2404.484751] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2418.731099] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2426.188817] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2454.206204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.408336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2454.450756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2454.525384] Lustre: lustre-MDT0000: new disk, initializing [ 2454.592740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.617732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2458.634962] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2467.574371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2467.678760] Lustre: 119294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2467.717439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2467.723771] Lustre: Skipped 1 previous similar message [ 2467.776894] Lustre: lustre-MDT0001: new disk, initializing [ 2467.868646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2467.900864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2467.905886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2471.467993] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2479.550611] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2484.337033] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2489.796046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2490.014849] Lustre: lustre-OST0000: new disk, initializing [ 2490.021367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2490.026567] Lustre: 121263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2490.112243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.194789] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2498.096728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2498.110460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2498.190167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2503.523514] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2507.518745] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2507.680679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2511.537923] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2511.711553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2513.381965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2513.386735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.407551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.601123] Lustre: server umount lustre-OST0000 complete [ 2524.128993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2524.132765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.133996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.134026] Lustre: Skipped 3 previous similar messages [ 2524.140303] LustreError: Skipped 1 previous similar message [ 2524.170917] Lustre: Skipped 2 previous similar messages [ 2529.329387] Lustre: server umount lustre-MDT0000 complete [ 2532.207524] LustreError: 119286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271733 with bad export cookie 16748455848933678371 [ 2532.211264] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.220483] LustreError: 119286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2532.461503] Lustre: server umount lustre-MDT0001 complete [ 2540.105401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.724499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.562625] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2546.145800] LustreError: 123479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.161219] LustreError: 123479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2551.278531] LustreError: 123478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.007720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.112922] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2566.008720] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2571.141281] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2578.920964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2587.509872] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2596.038229] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2607.169527] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2611.170352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2611.183573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.203593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.219498] Lustre: Skipped 3 previous similar messages [ 2617.638299] Lustre: server umount lustre-OST0000 complete [ 2624.994459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.023446] Lustre: Skipped 2 previous similar messages [ 2636.255263] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2636.472635] Lustre: server umount lustre-MDT0000 complete [ 2639.819167] LustreError: 123458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271841 with bad export cookie 16748455848933679253 [ 2639.826194] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.042458] Lustre: server umount lustre-MDT0001 complete [ 2649.175830] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2652.187951] Key type lgssc unregistered [ 2652.636675] LNet: 127109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.655596] LNetError: 127109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2652.681639] LNet: Removed LNI 192.168.203.138@tcp [ 2653.493129] Key type .llcrypt unregistered [ 2653.499542] Key type ._llcrypt unregistered [ 2669.808120] Key type ._llcrypt registered [ 2669.812184] Key type .llcrypt registered [ 2669.958220] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2683.479107] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2684.623209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2684.640388] alg: No test for adler32 (adler32-zlib) [ 2685.690342] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2685.929777] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2687.583169] Key type lgssc registered [ 2688.532801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2720.345045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2720.361740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.636620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2721.674617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2721.818521] Lustre: lustre-MDT0000: new disk, initializing [ 2721.895803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.911284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2726.225122] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2735.676544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.792369] Lustre: 130253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2735.837499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2735.843337] Lustre: Skipped 1 previous similar message [ 2735.930402] Lustre: lustre-MDT0001: new disk, initializing [ 2736.040855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.103501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2736.117397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2741.154429] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2750.812610] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2756.604211] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2763.290314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.625227] Lustre: lustre-OST0000: new disk, initializing [ 2763.630995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2763.637825] Lustre: 132222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2763.718551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.431164] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2772.595709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2772.609760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2772.722221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2778.498503] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2783.153661] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2783.335452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2787.699738] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2787.936947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2793.440296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.440721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.445172] LustreError: Skipped 1 previous similar message [ 2793.445887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.461136] Lustre: Skipped 1 previous similar message [ 2793.476885] Lustre: Skipped 1 previous similar message [ 2795.653749] Lustre: server umount lustre-OST0000 complete [ 2802.656607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2802.660756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.672379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.684329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.686035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.714551] Lustre: Skipped 1 previous similar message [ 2805.022851] Lustre: server umount lustre-MDT0000 complete [ 2808.225559] LustreError: 131049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272009 with bad export cookie 6758104980121741487 [ 2808.235351] LustreError: 131049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2808.239553] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.580776] Lustre: server umount lustre-MDT0001 complete [ 2816.454140] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:13:36 (1788272016) [ 2822.915332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.344331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.081514] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2828.775161] LustreError: 134632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.799310] LustreError: 134632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2832.865395] LustreError: 134633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.205711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.568441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2839.210871] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2846.020984] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2851.702709] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2855.904358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2855.907588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.919751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.739647] Lustre: server umount lustre-MDT0000 complete [ 2860.001495] LustreError: 134633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.021515] LustreError: 134633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2862.698161] LustreError: 134612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272064 with bad export cookie 6758104980121742369 [ 2862.707604] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2863.075840] Lustre: server umount lustre-MDT0001 complete [ 2869.776329] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:14:29 (1788272069) [ 2874.960640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.463835] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2895.583724] LustreError: 137046:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2921.311720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.696860] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2933.278768] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2935.035311] Lustre: server umount lustre-OST0000 complete [ 2942.171969] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:15:42 (1788272142) [ 2947.709749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.122230] LustreError: 138490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.205825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.907179] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2953.189256] LustreError: 138491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.232374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.032251] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2968.312129] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2972.207432] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2977.115740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.448783] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2988.323920] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2998.244772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.247116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.255291] Lustre: Skipped 2 previous similar messages [ 2998.258556] Lustre: Skipped 2 previous similar messages [ 3003.364301] LustreError: 140981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.379432] LustreError: 140981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3005.382629] Lustre: server umount lustre-MDT0000 complete [ 3007.886873] LustreError: 138471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272209 with bad export cookie 6758104980121743034 [ 3007.896036] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.899445] LustreError: 138471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3008.138880] Lustre: server umount lustre-MDT0001 complete [ 3019.748459] Lustre: server umount lustre-OST0000 complete [ 3026.186837] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:17:06 (1788272226) [ 3031.750844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.059838] LustreError: 142145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.116076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.125139] Lustre: Skipped 2 previous similar messages [ 3035.693380] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3041.648773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.196734] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3050.622653] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3054.782994] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3059.258605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3063.896123] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3069.911134] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3073.129729] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3077.868486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3078.556890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3081.270248] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3081.409908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3083.233187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.243456] Lustre: Skipped 1 previous similar message [ 3083.247731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.251344] Lustre: Skipped 3 previous similar messages [ 3084.842430] Lustre: server umount lustre-OST0000 complete [ 3087.840063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3093.862786] Lustre: server umount lustre-MDT0000 complete [ 3098.596151] LustreError: 142141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.630600] LustreError: 142141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3101.910966] LustreError: 142125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272303 with bad export cookie 6758104980121744105 [ 3101.921282] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.992219] LustreError: 142125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3102.532659] Lustre: server umount lustre-MDT0001 complete [ 3113.919196] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:18:33 (1788272313) [ 3123.975880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.615706] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3144.735441] LustreError: 146341:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 3170.463669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.470241] Lustre: Skipped 2 previous similar messages [ 3180.351343] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3189.988360] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3197.603707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.088388] LustreError: 146366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.292591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3203.520284] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3212.685461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.641498] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3229.113733] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3236.262985] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3242.525110] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3242.836390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3247.561733] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3247.821192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3252.192963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.209724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.213773] Lustre: Skipped 5 previous similar messages [ 3252.227278] Lustre: Skipped 7 previous similar messages [ 3255.986352] Lustre: server umount lustre-MDT0000 complete [ 3259.630045] LustreError: 146347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272461 with bad export cookie 6758104980121745127 [ 3259.645389] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.651034] LustreError: 146347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3259.973809] Lustre: server umount lustre-MDT0001 complete [ 3273.876293] Lustre: server umount lustre-OST0000 complete [ 3282.369050] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:21:22 (1788272482) [ 3289.438492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.991780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3310.113502] LustreError: 150537:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 3335.903295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.915035] Lustre: Skipped 2 previous similar messages [ 3342.418437] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3350.098184] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3356.719471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.025378] LustreError: 150562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.050339] LustreError: 150562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3357.188573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3361.242654] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3369.243129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.100040] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3381.728443] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3387.676901] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3395.132028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.302382] Lustre: lustre-OST0001: new disk, initializing [ 3395.305780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3395.310917] Lustre: 153411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3400.751134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3400.759041] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3400.873924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3401.199602] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3410.995994] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3415.467248] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3415.644532] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3420.611610] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3420.823676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3423.714987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.721976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.740065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.517073] Lustre: server umount lustre-OST0000 complete [ 3436.515129] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.522192] LustreError: Skipped 1 previous similar message [ 3439.422585] Lustre: server umount lustre-OST0001 complete [ 3449.855099] Lustre: server umount lustre-MDT0000 complete [ 3453.833616] LustreError: 150543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272655 with bad export cookie 6758104980121746079 [ 3453.841603] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3453.850108] LustreError: 150543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3454.241769] Lustre: server umount lustre-MDT0001 complete [ 3477.266369] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:24:37 (1788272677) [ 3478.535941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3480.003720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3481.471314] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:24:41 (1788272681) [ 3487.493106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.724775] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3487.760328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3492.568250] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3501.234596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.314669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3501.346967] Lustre: 156971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.154066] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3513.615469] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3519.577929] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3526.101829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.384592] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3532.976398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3533.572869] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3541.344206] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3545.923070] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3546.216248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3550.996287] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3551.298855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3553.762665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.769986] LustreError: Skipped 2 previous similar messages [ 3553.773649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.788065] Lustre: Skipped 5 previous similar messages [ 3553.795431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.798076] Lustre: Skipped 8 previous similar messages [ 3559.508226] Lustre: server umount lustre-OST0000 complete [ 3581.237216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.657759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3583.675022] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3588.166196] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3595.720779] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3599.024363] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3599.041252] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3600.698070] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3600.936336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3605.689657] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3605.958496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3612.252758] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fb8cbe41000.ost_server_uuid 50 [ 3613.853663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9fb8cbe41000.ost_server_uuid in FULL state after 0 sec [ 3614.088249] LustreError: 156977:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.38@tcp) returned error from blocking AST (req@00000000cc3c2553 x1875138982438656 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000550d25c1/0x5dc99cac66f483f9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.38@tcp remote: 0xff5b4a931355d1b9 expref: 5 pid: 159572 timeout: 3714 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3614.145305] LustreError: lustre-MDT0000: A client on nid 192.168.203.38@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3614.160409] LustreError: 156967:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.38@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000550d25c1/0x5dc99cac66f483f9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.38@tcp remote: 0xff5b4a931355d1b9 expref: 6 pid: 159572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3619.809172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3619.822074] LustreError: Skipped 2 previous similar messages [ 3624.387711] Lustre: server umount lustre-OST0000 complete [ 3642.335152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3642.533793] Lustre: server umount lustre-MDT0000 complete [ 3645.409743] LustreError: 156978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.423732] LustreError: 156978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3646.174652] LustreError: 157714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272847 with bad export cookie 6758104980121747262 [ 3646.182151] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.187088] LustreError: 157714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3652.507209] Lustre: server umount lustre-MDT0001 complete [ 3663.795626] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3666.983578] Key type lgssc unregistered [ 3667.353395] LNet: 163085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3667.361127] LNetError: 163085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3667.377950] LNet: Removed LNI 192.168.203.138@tcp [ 3668.307342] Key type .llcrypt unregistered [ 3668.308813] Key type ._llcrypt unregistered [ 3694.142459] Key type ._llcrypt registered [ 3694.144966] Key type .llcrypt registered [ 3694.274561] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3695.230569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3695.327856] alg: No test for adler32 (adler32-zlib) [ 3696.441955] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 3696.689220] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3698.392604] Key type lgssc registered [ 3699.550691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3707.774582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3707.801632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.372781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.935532] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3718.631552] LustreError: 164501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.646773] LustreError: 164501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3722.728395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.016817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3727.615288] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3734.388480] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3739.855471] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3746.382776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3746.715498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.427886] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3753.923979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3759.260394] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3764.712785] Lustre: Failing over lustre-MDT0000 [ 3765.013476] Lustre: server umount lustre-MDT0000 complete [ 3766.385433] LustreError: 166415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.314453] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.319793] LustreError: 164497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.319821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3769.338626] Lustre: Skipped 2 previous similar messages [ 3769.373941] LustreError: 164497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3774.435819] LustreError: 164502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3774.457606] LustreError: 164502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3776.954692] LustreError: 164481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.38@tcp arrived at 1788272978 with bad export cookie 15996002061156746948 [ 3776.973405] LustreError: 164481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3779.553738] LustreError: 166414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.604400] LustreError: 166414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3784.671426] Lustre: 163813:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788272970/real 1788272970] req@00000000cb3fb93f x1875140042320896/t0(0) o400->MGC192.168.203.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788272986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3784.703597] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3785.654212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.794418] LustreError: 164502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.808244] LustreError: 164502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3795.286559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.377318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3798.747039] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3800.555160] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3800.560357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3801.373937] Lustre: lustre-MDT0000: Denying connection for new client 95d1928c-dfa9-41cf-80bf-5e2302784769 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 3810.786650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.804723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3810.806122] Lustre: Skipped 1 previous similar message [ 3815.924651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.946424] Lustre: Skipped 2 previous similar messages [ 3821.027953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.035644] Lustre: Skipped 1 previous similar message [ 3824.095188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3824.377753] Lustre: server umount lustre-OST0000 complete [ 3828.653835] LustreError: 168236:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3828.674224] Lustre: 167585:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3828.682840] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3828.717841] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3828.741241] LustreError: 167585:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009727c8ec x1875140042342656/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3828.769931] LustreError: 167585:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3828.781194] LustreError: 167585:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3828.802446] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 3829.323606] Lustre: server umount lustre-MDT0000 complete [ 3831.264384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.279951] LustreError: 164498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.306543] LustreError: 164498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3833.117067] LustreError: 164482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273034 with bad export cookie 15996002061156747046 [ 3833.117547] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.136389] LustreError: 164482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3833.169439] LustreError: 163810:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3835.470599] Lustre: server umount lustre-MDT0001 complete [ 3848.986697] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3851.625789] Key type lgssc unregistered [ 3851.922902] LNet: 168992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3851.945862] LNetError: 168992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.975377] LNet: Removed LNI 192.168.203.138@tcp [ 3852.835095] Key type .llcrypt unregistered [ 3852.840630] Key type ._llcrypt unregistered [ 3877.658740] Key type ._llcrypt registered [ 3877.661214] Key type .llcrypt registered [ 3877.768494] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3878.711316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3878.774459] alg: No test for adler32 (adler32-zlib) [ 3879.887739] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 3880.229822] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3881.992313] Key type lgssc registered [ 3883.320725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.726109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.759118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.279406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.245969] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3903.460045] LustreError: 170404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.484371] LustreError: 170404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3907.277588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.549884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.593802] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3918.676717] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3923.580778] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3929.520357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3929.951433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.346676] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3938.169707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3946.416468] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3953.634093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.644371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.678957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3953.689203] Lustre: Skipped 1 previous similar message [ 3955.653681] Lustre: server umount lustre-OST0000 complete [ 3963.872744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.874770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3963.875849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3963.886267] Lustre: Skipped 2 previous similar messages [ 3965.734563] Lustre: server umount lustre-MDT0000 complete [ 3969.004213] LustreError: 171132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.024187] LustreError: 171132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3969.904100] LustreError: 170383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273171 with bad export cookie 10247565490428488339 [ 3969.926015] LustreError: 170383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3969.936471] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.241634] Lustre: server umount lustre-MDT0001 complete [ 3981.341509] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3984.472457] Key type lgssc unregistered [ 3984.776873] LNet: 173938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3984.782166] LNetError: 173938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3984.798355] LNet: Removed LNI 192.168.203.138@tcp [ 3985.675162] Key type .llcrypt unregistered [ 3985.680032] Key type ._llcrypt unregistered [ 4020.329319] Key type ._llcrypt registered [ 4020.336463] Key type .llcrypt registered [ 4020.465487] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4021.564210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4021.666472] alg: No test for adler32 (adler32-zlib) [ 4022.676461] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4022.944051] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4024.616110] Key type lgssc registered [ 4025.614616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4032.581585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4032.612688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.068437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.625391] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4044.260196] LustreError: 175640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.289106] LustreError: 175640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4046.418758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.743924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4051.252605] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4058.534995] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4064.521713] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4071.548563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.877592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.407864] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4079.029318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4084.470416] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4093.415734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4093.426211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.444851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4093.926620] Lustre: server umount lustre-OST0000 complete [ 4098.022686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4098.033152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.053000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.552832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.567333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.588273] Lustre: Skipped 1 previous similar message [ 4104.674840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.679434] Lustre: Skipped 1 previous similar message [ 4107.823815] Lustre: server umount lustre-MDT0000 complete [ 4108.784806] LustreError: 178082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.818908] LustreError: 178082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4111.818894] LustreError: 175622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273313 with bad export cookie 1039501456505890335 [ 4111.828603] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.847024] LustreError: 175622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4112.342193] Lustre: server umount lustre-MDT0001 complete [ 4124.545254] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4127.144929] Key type lgssc unregistered [ 4127.437503] LNet: 179177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4127.446725] LNetError: 179177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4127.474564] LNet: Removed LNI 192.168.203.138@tcp [ 4128.256515] Key type .llcrypt unregistered [ 4128.261238] Key type ._llcrypt unregistered [ 4148.201380] Key type ._llcrypt registered [ 4148.203626] Key type .llcrypt registered [ 4148.296663] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4149.289590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4149.359374] alg: No test for adler32 (adler32-zlib) [ 4150.442837] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4150.694886] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4152.359359] Key type lgssc registered [ 4153.473968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4163.838522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4163.852463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.194776] Lustre: *** cfs_fail_loc=135, val=0*** [ 4165.294571] LustreError: 180689:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4165.305032] LustreError: 180689:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -2 [ 4165.314342] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4165.314342] [ 4165.323514] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4165.339386] LustreError: 180650:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4165.348509] LustreError: 180650:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 4165.365663] LustreError: 180650:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 4165.374869] LustreError: 180650:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4165.446931] Lustre: server umount lustre-MDT0000 complete [ 4165.449801] LustreError: 180650:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4171.531763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.952755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.670414] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4182.504274] LustreError: 181060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.525480] LustreError: 181060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4183.642463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.035701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.960889] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4196.835722] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4202.641553] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4207.590535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.594013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.601548] Lustre: Skipped 1 previous similar message [ 4207.625521] Lustre: Skipped 1 previous similar message [ 4208.609621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.618716] Lustre: Skipped 1 previous similar message [ 4213.733258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.850370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.859704] Lustre: Skipped 2 previous similar messages [ 4219.359149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4219.509807] Lustre: server umount lustre-MDT0000 complete [ 4222.908405] LustreError: 181042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273424 with bad export cookie 10763219502088897819 [ 4222.912595] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.924036] LustreError: 181042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4223.549338] Lustre: server umount lustre-MDT0001 complete [ 4232.501059] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4235.481087] Key type lgssc unregistered [ 4235.851310] LNet: 183385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4235.873744] LNetError: 183385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4235.898744] LNet: Removed LNI 192.168.203.138@tcp [ 4236.706417] Key type .llcrypt unregistered [ 4236.708730] Key type ._llcrypt unregistered [ 4255.972929] Key type ._llcrypt registered [ 4255.975509] Key type .llcrypt registered [ 4256.066939] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4256.499384] Key type .llcrypt unregistered [ 4256.501495] Key type ._llcrypt unregistered [ 4272.468120] Key type ._llcrypt registered [ 4272.469948] Key type .llcrypt registered [ 4272.571558] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4273.560090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4273.576340] alg: No test for adler32 (adler32-zlib) [ 4274.592785] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4274.821329] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4276.463444] Key type lgssc registered [ 4277.631631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4285.665061] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4285.683354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.279430] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4306.399348] LustreError: 185520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 4333.023436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.954595] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4344.579487] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4350.403394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.921995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.990957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4354.889544] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4356.068306] LustreError: 185546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.077655] LustreError: 185546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4361.189319] LustreError: 185545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.280769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.606628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4366.636528] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4373.634766] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4380.260788] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4383.545290] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4397.030123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4397.045244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.070946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.563720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.568806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.575422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.672970] Lustre: server umount lustre-OST0000 complete [ 4408.801404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.804161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4408.805104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.805109] Lustre: Skipped 1 previous similar message [ 4408.819052] Lustre: Skipped 1 previous similar message [ 4411.591965] Lustre: server umount lustre-MDT0000 complete [ 4413.927429] LustreError: 186561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.950256] LustreError: 186561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4415.244492] LustreError: 185528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273616 with bad export cookie 13093875007056201175 [ 4415.249323] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.260471] LustreError: 185528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4415.599815] Lustre: server umount lustre-MDT0001 complete [ 4425.071479] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4427.026744] Key type lgssc unregistered [ 4427.275334] LNet: 189488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4427.284408] LNetError: 189488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4427.306771] LNet: Removed LNI 192.168.203.138@tcp [ 4427.978581] Key type .llcrypt unregistered [ 4427.980670] Key type ._llcrypt unregistered [ 4448.188952] Key type ._llcrypt registered [ 4448.192428] Key type .llcrypt registered [ 4448.263761] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4449.029292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4449.067656] alg: No test for adler32 (adler32-zlib) [ 4450.139848] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4450.350714] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4452.016491] Key type lgssc registered [ 4453.031734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4459.336927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4459.351919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.705588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.647892] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4471.266506] LustreError: 190881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.288267] LustreError: 190881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4471.406831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.714258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4475.022041] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4481.531402] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4485.854874] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4490.783090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4491.114376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.770304] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4503.104839] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.478644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4511.851055] Lustre: Failing over lustre-MDT0000 [ 4512.069478] Lustre: server umount lustre-MDT0000 complete [ 4512.736131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4512.745309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.757757] LustreError: 190880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.882849] LustreError: 190874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.834484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.853467] Lustre: Skipped 1 previous similar message [ 4519.981946] LustreError: 192361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.995031] LustreError: 192361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4525.103284] LustreError: 193086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.126368] LustreError: 193086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4528.920208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.041760] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4529.309185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.348082] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4530.228457] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4533.206744] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4534.767281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4534.790363] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4534.838229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4537.218309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4538.885391] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4548.576650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4548.582154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.592058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.115720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.126472] Lustre: Skipped 1 previous similar message [ 4550.297024] Lustre: server umount lustre-OST0000 complete [ 4555.233835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4555.234397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.245467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4555.255424] Lustre: Skipped 2 previous similar messages [ 4555.272171] Lustre: Skipped 1 previous similar message [ 4559.194524] Lustre: server umount lustre-MDT0000 complete [ 4560.355766] LustreError: 190881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.367583] LustreError: 190881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4561.755622] LustreError: 190861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273763 with bad export cookie 4065966946890390282 [ 4561.762236] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.771637] LustreError: 190861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4562.014553] Lustre: server umount lustre-MDT0001 complete [ 4570.199101] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4572.328610] Key type lgssc unregistered [ 4572.544204] LNet: 195621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4572.550353] LNetError: 195621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4572.561148] LNet: Removed LNI 192.168.203.138@tcp [ 4573.170857] Key type .llcrypt unregistered [ 4573.173304] Key type ._llcrypt unregistered [ 4593.520647] Key type ._llcrypt registered [ 4593.523451] Key type .llcrypt registered [ 4593.604580] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4594.385898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4594.421300] alg: No test for adler32 (adler32-zlib) [ 4595.369349] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4595.577060] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4597.223151] Key type lgssc registered [ 4598.024433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4603.228547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4603.252731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.557108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.633757] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4613.274898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.540539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4616.211889] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4620.996468] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4626.000441] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4631.651469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4631.966436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.211985] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4642.831160] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4649.389560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4650.464814] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4653.383355] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4667.205061] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4679.136552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4679.153810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.167926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.161867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.173575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.186481] Lustre: Skipped 1 previous similar message [ 4681.633882] Lustre: server umount lustre-OST0000 complete [ 4685.286968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4685.292839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.307230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4685.311765] Lustre: Skipped 1 previous similar message [ 4690.379622] Lustre: server umount lustre-MDT0000 complete [ 4690.400838] LustreError: 197032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4690.411587] LustreError: 197032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4692.840927] LustreError: 197010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273894 with bad export cookie 17248737058710672590 [ 4692.845286] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.848725] LustreError: 197010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4693.072185] Lustre: server umount lustre-MDT0001 complete [ 4700.118362] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4702.204727] Key type lgssc unregistered [ 4702.427858] LNet: 200895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4702.435745] LNetError: 200895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4702.456270] LNet: Removed LNI 192.168.203.138@tcp [ 4703.071905] Key type .llcrypt unregistered [ 4703.075700] Key type ._llcrypt unregistered [ 4717.310941] Key type ._llcrypt registered [ 4717.313895] Key type .llcrypt registered [ 4717.417867] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4718.265836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4718.281849] alg: No test for adler32 (adler32-zlib) [ 4719.263283] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4719.465426] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4721.103202] Key type lgssc registered [ 4722.021513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4727.340137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4727.350509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.653767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.770661] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4737.561139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.847367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4740.996624] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4745.834196] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.411931] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4753.884683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.123083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.186953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4757.820023] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4762.733858] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4766.728746] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4774.210498] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4780.322845] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4788.806886] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4799.460054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.461742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4799.462309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4799.473178] Lustre: Skipped 1 previous similar message [ 4804.578669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4804.583261] Lustre: Skipped 2 previous similar messages [ 4809.696988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4809.701957] Lustre: Skipped 1 previous similar message [ 4811.231294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4811.331874] Lustre: server umount lustre-OST0000 complete [ 4814.817644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.817818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4814.821942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.826984] Lustre: Skipped 1 previous similar message [ 4819.936414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.942613] Lustre: Skipped 2 previous similar messages [ 4828.128468] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4828.254239] Lustre: server umount lustre-MDT0000 complete [ 4830.175971] LustreError: 202948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.187273] LustreError: 202948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4830.582953] LustreError: 202122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274031 with bad export cookie 12809961545495227551 [ 4830.587123] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.593562] LustreError: 202122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4830.812418] Lustre: server umount lustre-MDT0001 complete [ 4837.648896] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4839.757199] Key type lgssc unregistered [ 4840.019903] LNet: 207160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4840.029532] LNetError: 207160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4840.050402] LNet: Removed LNI 192.168.203.138@tcp [ 4840.700243] Key type .llcrypt unregistered [ 4840.703437] Key type ._llcrypt unregistered [ 4855.624706] Key type ._llcrypt registered [ 4855.625838] Key type .llcrypt registered [ 4855.688051] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4856.449489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4856.486172] alg: No test for adler32 (adler32-zlib) [ 4857.411035] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4857.569910] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4859.207334] Key type lgssc registered [ 4859.995697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4865.061675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4865.077246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4866.422148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.060501] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4874.352418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.585270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4877.184639] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4882.157630] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4885.939624] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4890.415742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4890.664475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.691930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4894.050955] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4898.500672] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.026722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4909.039340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4909.042549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4909.044122] Lustre: Skipped 1 previous similar message [ 4914.150271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4914.171848] Lustre: Skipped 2 previous similar messages [ 4915.329521] Lustre: server umount lustre-OST0000 complete [ 4924.385266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.392746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4924.399236] Lustre: Skipped 1 previous similar message [ 4929.511623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4929.524445] Lustre: Skipped 1 previous similar message [ 4929.643757] Lustre: server umount lustre-MDT0000 complete [ 4934.650863] LustreError: 209370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.706305] LustreError: 209370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4936.223663] LustreError: 208533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274137 with bad export cookie 11069832639923228834 [ 4936.241262] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.257925] LustreError: 208533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4936.975457] Lustre: server umount lustre-MDT0001 complete [ 4953.869585] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4957.447705] Key type lgssc unregistered [ 4957.817889] LNet: 212241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4957.823622] LNetError: 212241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4957.842353] LNet: Removed LNI 192.168.203.138@tcp [ 4958.740788] Key type .llcrypt unregistered [ 4958.744156] Key type ._llcrypt unregistered [ 4976.954124] Key type ._llcrypt registered [ 4976.957480] Key type .llcrypt registered [ 4977.075415] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4978.421976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.459613] alg: No test for adler32 (adler32-zlib) [ 4979.503527] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 4979.729518] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4981.423138] Key type lgssc registered [ 4982.876657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4990.622308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4990.651553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.136652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.966159] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5001.697331] LustreError: 213488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.710499] LustreError: 213488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5003.686362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.000365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5007.834731] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5014.848306] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5020.009738] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5026.137979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.449224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.723233] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5036.742183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5039.627397] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5052.388857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.397436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.414982] Lustre: Skipped 1 previous similar message [ 5057.322481] Lustre: server umount lustre-OST0000 complete [ 5060.580996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5060.586162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.606726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5060.620033] Lustre: Skipped 1 previous similar message [ 5062.627927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.628449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5066.870479] Lustre: server umount lustre-MDT0000 complete [ 5067.745827] LustreError: 213489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.763519] LustreError: 213489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5070.480980] LustreError: 213467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274271 with bad export cookie 1941833796645401428 [ 5070.491184] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.497400] LustreError: 213467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5070.899197] Lustre: server umount lustre-MDT0001 complete [ 5081.056672] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 5083.967547] Key type lgssc unregistered [ 5084.258281] LNet: 217173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5084.266720] LNetError: 217173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5084.293900] LNet: Removed LNI 192.168.203.138@tcp [ 5085.230210] Key type .llcrypt unregistered [ 5085.236454] Key type ._llcrypt unregistered [ 5100.551270] Key type ._llcrypt registered [ 5100.554958] Key type .llcrypt registered [ 5100.645868] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5101.570690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5101.615536] alg: No test for adler32 (adler32-zlib) [ 5102.657917] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5102.908828] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 5104.639534] Key type lgssc registered [ 5105.635143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5111.932563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5111.951243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.366267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5117.054917] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5123.553180] LustreError: 218418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.574458] LustreError: 218418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5124.328933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.610525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5127.844688] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5134.123942] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5139.529800] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5146.507939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.006626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5153.108215] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5158.273381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5159.391173] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5168.878324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5169.031759] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5169.405925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5175.922416] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5184.249145] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5185.624345] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5197.243825] Lustre: Permanently deactivating lustre-OST0001 [ 5197.267707] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5204.094958] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5212.233100] Lustre: Permanently reactivating lustre-OST0001 [ 5212.271939] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5212.280621] Lustre: Skipped 2 previous similar messages [ 5213.368676] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.384215] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5213.390357] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5213.401976] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5213.431800] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5223.917618] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5223.918457] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.923129] LustreError: Skipped 1 previous similar message [ 5223.930503] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5223.950828] Lustre: Skipped 2 previous similar messages [ 5223.970347] Lustre: Skipped 1 previous similar message [ 5224.978917] Lustre: server umount lustre-OST0001 complete [ 5229.545243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.559728] LustreError: 220237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.564574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.577636] LustreError: 220237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5229.587311] Lustre: Skipped 3 previous similar messages [ 5234.668155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.669468] LustreError: 220236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.685347] Lustre: Skipped 1 previous similar message [ 5234.705721] LustreError: 220236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5236.138571] Lustre: server umount lustre-OST0000 complete [ 5242.337090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5242.349256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5242.363504] Lustre: Skipped 1 previous similar message [ 5242.367980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.525616] LustreError: 222713:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c14e6df7) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5246.694271] Lustre: server umount lustre-MDT0000 complete [ 5250.025538] LustreError: 218413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5250.053657] LustreError: 218413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5251.161699] LustreError: 218397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274452 with bad export cookie 15523349098281248907 [ 5251.167154] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5251.173187] LustreError: 218397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5251.501082] Lustre: server umount lustre-MDT0001 complete [ 5261.981913] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 5264.256636] Key type lgssc unregistered [ 5264.476071] LNet: 223469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5264.485633] LNetError: 223469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5264.500691] LNet: Removed LNI 192.168.203.138@tcp [ 5265.195340] Key type .llcrypt unregistered [ 5265.202883] Key type ._llcrypt unregistered [ 5306.454638] Key type ._llcrypt registered [ 5306.458475] Key type .llcrypt registered [ 5306.581586] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5307.623078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5307.649190] alg: No test for adler32 (adler32-zlib) [ 5308.780621] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5309.006519] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 5310.655200] Key type lgssc registered [ 5312.064940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5320.338378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5320.360789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.810490] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5321.834698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5321.959225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.563055] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5337.212222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.385108] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5337.469077] Lustre: 225674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5337.857650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.338513] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5349.956406] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5357.042280] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5366.150092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5366.351200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5366.606636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.080687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5374.282163] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5382.138954] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5386.008807] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5390.934119] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5400.176451] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5406.526810] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5417.307588] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5437.588219] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5437.595875] Lustre: Skipped 2 previous similar messages [ 5473.195573] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5473.207251] Lustre: Skipped 3 previous similar messages [ 5542.169508] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5542.174594] Lustre: Skipped 8 previous similar messages [ 5558.240768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.247651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.275120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.758593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.770572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.790387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.598911] Lustre: server umount lustre-OST0000 complete [ 5568.483476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5568.492483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.514604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.119339] Lustre: server umount lustre-MDT0000 complete [ 5578.905554] LustreError: 225664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274780 with bad export cookie 11523123794453140109 [ 5578.917549] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5579.363690] Lustre: server umount lustre-MDT0001 complete [ 5596.167796] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 5599.212415] Key type lgssc unregistered [ 5599.809718] LNet: 230502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.828520] LNetError: 230502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5599.847396] LNet: Removed LNI 192.168.203.138@tcp [ 5600.516628] Key type .llcrypt unregistered [ 5600.518981] Key type ._llcrypt unregistered [ 5626.623786] Key type ._llcrypt registered [ 5626.630177] Key type .llcrypt registered [ 5626.727670] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5627.562996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5627.608204] alg: No test for adler32 (adler32-zlib) [ 5628.889195] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5629.256113] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 5630.951152] Key type lgssc registered [ 5632.214564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5640.426741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5640.450587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.829814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.324645] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5652.453786] LustreError: 231915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.475591] LustreError: 231915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5654.827434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.151965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5659.749907] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5667.399244] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5673.300147] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5679.696488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5680.063581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5686.522519] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5687.287905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5693.778178] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5715.939633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5715.950070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.970752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5717.984657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.990972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5718.004204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.035415] Lustre: Skipped 1 previous similar message [ 5719.725932] Lustre: server umount lustre-OST0000 complete [ 5726.689879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5726.696223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.712887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.065112] Lustre: server umount lustre-MDT0000 complete [ 5733.233535] LustreError: 232649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274934 with bad export cookie 18206450554275145711 [ 5733.240121] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.240429] LustreError: 232649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5733.613470] Lustre: server umount lustre-MDT0001 complete [ 5743.180309] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 5745.469658] Key type lgssc unregistered [ 5745.735933] LNet: 235616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5745.744483] LNetError: 235616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5745.762534] LNet: Removed LNI 192.168.203.138@tcp [ 5746.407478] Key type .llcrypt unregistered [ 5746.410230] Key type ._llcrypt unregistered [ 5774.145313] Key type ._llcrypt registered [ 5774.147929] Key type .llcrypt registered [ 5774.373660] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5775.146085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.311522] alg: No test for adler32 (adler32-zlib) [ 5776.399964] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5776.583874] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 5778.207155] Key type lgssc registered [ 5779.572512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6048.255501] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 6050.879503] Key type lgssc unregistered [ 6051.197762] LNet: 237640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6051.206083] LNetError: 237640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6051.226185] LNet: Removed LNI 192.168.203.138@tcp [ 6051.793426] Key type .llcrypt unregistered [ 6051.796579] Key type ._llcrypt unregistered [ 6082.638536] Key type ._llcrypt registered [ 6082.640980] Key type .llcrypt registered [ 6082.790744] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 6101.368833] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6102.398760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6102.616044] alg: No test for adler32 (adler32-zlib) [ 6103.670094] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 6103.873095] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 6105.511214] Key type lgssc registered [ 6106.836190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6133.771179] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5939 sec ========== 11:08:53 (1788275333) [ 6135.161528] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:08:55 (1788275335) === [ 6137.893155] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:08:57 (1788275337) === [ 6178.736508] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 6181.528398] Key type lgssc unregistered [ 6181.804433] LNet: 242879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6181.816130] LNetError: 242879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6181.837367] LNet: Removed LNI 192.168.203.138@tcp [ 6182.669212] Key type .llcrypt unregistered [ 6182.671986] Key type ._llcrypt unregistered