[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476977028 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001024] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009135] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012062] *** VALIDATE selinux *** [ 0.020717] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025394] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028051] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] *** VALIDATE tmpfs *** [ 0.031221] *** VALIDATE proc *** [ 0.032240] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035083] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.041925] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044636] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045035] ... version: 2 [ 0.046024] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050024] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052263] rcu: Hierarchical SRCU implementation. [ 0.054256] smp: Bringing up secondary CPUs ... [ 0.055609] x86: Booting SMP configuration: [ 0.056032] .... node #0, CPUs: #1 #2 #3 [ 0.059418] smp: Brought up 1 node, 4 CPUs [ 0.061042] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161352] node 0 deferred pages initialised in 97ms [ 0.165009] devtmpfs: initialized [ 0.166297] x86/mm: Memory block size: 128MB [ 0.169880] gcov: version magic: 0x41383552 [ 0.174289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.179236] pinctrl core: initialized pinctrl subsystem [ 0.181182] [ 0.181675] ************************************************************* [ 0.184013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186011] ** ** [ 0.189014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.191012] ** ** [ 0.193015] ** This means that this kernel is built to expose internal ** [ 0.196012] ** IOMMU data structures, which may compromise security on ** [ 0.198010] ** your system. ** [ 0.200014] ** ** [ 0.203013] ** If you see this message and you are not debugging the ** [ 0.205011] ** kernel, report this immediately to your vendor! ** [ 0.207014] ** ** [ 0.210014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212011] ************************************************************* [ 0.215638] NET: Registered protocol family 16 [ 0.216347] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.219053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.221057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225035] cpuidle: using governor menu [ 0.227185] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228427] PCI: Using configuration type 1 for base access [ 0.230124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239044] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244130] cryptd: max_cpu_qlen set to 1000 [ 0.247271] ACPI: Added _OSI(Module Device) [ 0.248015] ACPI: Added _OSI(Processor Device) [ 0.250024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255740] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262278] ACPI: Interpreter enabled [ 0.263055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265012] ACPI: Using IOAPIC for interrupt routing [ 0.267097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279403] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.283015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.288979] acpiphp: Slot [2] registered [ 0.291108] acpiphp: Slot [5] registered [ 0.292078] acpiphp: Slot [6] registered [ 0.293109] acpiphp: Slot [7] registered [ 0.295130] acpiphp: Slot [8] registered [ 0.296113] acpiphp: Slot [9] registered [ 0.298120] acpiphp: Slot [10] registered [ 0.299123] acpiphp: Slot [3] registered [ 0.300089] acpiphp: Slot [4] registered [ 0.302075] acpiphp: Slot [11] registered [ 0.303062] acpiphp: Slot [12] registered [ 0.304132] acpiphp: Slot [13] registered [ 0.305060] acpiphp: Slot [14] registered [ 0.307089] acpiphp: Slot [15] registered [ 0.308052] acpiphp: Slot [16] registered [ 0.308989] acpiphp: Slot [17] registered [ 0.310083] acpiphp: Slot [18] registered [ 0.311088] acpiphp: Slot [19] registered [ 0.312115] acpiphp: Slot [20] registered [ 0.314089] acpiphp: Slot [21] registered [ 0.315071] acpiphp: Slot [22] registered [ 0.316081] acpiphp: Slot [23] registered [ 0.317093] acpiphp: Slot [24] registered [ 0.319093] acpiphp: Slot [25] registered [ 0.320116] acpiphp: Slot [26] registered [ 0.321097] acpiphp: Slot [27] registered [ 0.323111] acpiphp: Slot [28] registered [ 0.324124] acpiphp: Slot [29] registered [ 0.326104] acpiphp: Slot [30] registered [ 0.327087] acpiphp: Slot [31] registered [ 0.329081] PCI host bridge to bus 0000:00 [ 0.330018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345964] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349290] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357570] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373182] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376880] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382907] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414050] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467251] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.484015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.516605] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.523017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.549015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.560046] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.574020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.578021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604521] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.617020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.631019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641112] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.654140] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673107] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687332] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692491] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695430] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.700288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705029] iommu: Default domain type: Passthrough [ 0.706397] SCSI subsystem initialized [ 0.708165] ACPI: bus type USB registered [ 0.709111] usbcore: registered new interface driver usbfs [ 0.711078] usbcore: registered new interface driver hub [ 0.713091] usbcore: registered new device driver usb [ 0.715184] pps_core: LinuxPPS API ver. 1 registered [ 0.717028] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721088] PTP clock support registered [ 0.724056] EDAC MC: Ver: 3.0.0 [ 0.725452] PCI: Using ACPI for IRQ routing [ 0.726809] NetLabel: Initializing [ 0.727010] NetLabel: domain hash size = 128 [ 0.728014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729193] NetLabel: unlabeled traffic allowed by default [ 0.730382] vgaarb: loaded [ 0.732273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.734019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741000] clocksource: Switched to clocksource kvm-clock [ 0.849126] VFS: Disk quotas dquot_6.6.0 [ 0.850347] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.852479] *** VALIDATE ramfs *** [ 0.853432] *** VALIDATE hugetlbfs *** [ 0.855033] pnp: PnP ACPI init [ 0.856863] pnp: PnP ACPI: found 6 devices [ 0.873803] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.876949] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.878972] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.881232] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.883686] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.885788] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.887811] NET: Registered protocol family 2 [ 0.889822] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.894530] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.897430] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.901885] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.905403] TCP: Hash tables configured (established 65536 bind 65536) [ 0.907620] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.910335] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.912290] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914431] NET: Registered protocol family 1 [ 0.918105] RPC: Registered named UNIX socket transport module. [ 0.920098] RPC: Registered udp transport module. [ 0.921937] RPC: Registered tcp transport module. [ 0.923478] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.926092] NET: Registered protocol family 44 [ 0.927920] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.930110] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.932255] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.934614] PCI: CLS 0 bytes, default 64 [ 0.936939] Unpacking initramfs... [ 2.344752] debug: unmapping init [mem 0xffff9b1c3cc54000-0xffff9b1c3ffbffff] [ 2.348835] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.351282] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.354365] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.871966] Initialise system trusted keyrings [ 2.873932] Key type blacklist registered [ 2.876124] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887254] zbud: loaded [ 2.891096] *** VALIDATE nfs *** [ 2.892442] *** VALIDATE nfs4 *** [ 2.894466] pstore: using deflate compression [ 2.897582] Platform Keyring initialized [ 3.019461] NET: Registered protocol family 38 [ 3.020906] Key type asymmetric registered [ 3.022122] Asymmetric key parser 'x509' registered [ 3.023586] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.026126] io scheduler mq-deadline registered [ 3.027588] io scheduler kyber registered [ 3.029174] io scheduler bfq registered [ 3.030949] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.033446] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.035893] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.038230] ACPI: Power Button [PWRF] [ 3.043169] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.049631] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.068344] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.076752] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.090663] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.121886] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152450] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.157398] Non-volatile memory driver v1.3 [ 3.159153] Linux agpgart interface v0.103 [ 3.196043] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.200220] vda: detected capacity change from 0 to 75194368 [ 3.215993] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.219379] vdb: detected capacity change from 0 to 1073741824 [ 3.233828] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237427] vdc: detected capacity change from 0 to 2621440000 [ 3.257865] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.261273] vdd: detected capacity change from 0 to 2621440000 [ 3.278104] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.281314] vde: detected capacity change from 0 to 4294967296 [ 3.298892] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.301947] vdf: detected capacity change from 0 to 4294967296 [ 3.309866] libphy: Fixed MDIO Bus: probed [ 3.314799] usbcore: registered new interface driver usbserial_generic [ 3.317437] usbserial: USB Serial support registered for generic [ 3.319936] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.324516] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.326328] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.329100] mousedev: PS/2 mouse device common for all mice [ 3.332095] rtc_cmos 00:05: RTC can wake from S4 [ 3.334815] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.336925] rtc_cmos 00:05: registered as rtc0 [ 3.342528] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.344298] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.346230] intel_pstate: CPU model not supported [ 3.352679] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.356837] hid: raw HID events driver (C) Jiri Kosina [ 3.359300] usbcore: registered new interface driver usbhid [ 3.361743] usbhid: USB HID core driver [ 3.363538] drop_monitor: Initializing network drop monitor service [ 3.366829] Initializing XFRM netlink socket [ 3.369241] NET: Registered protocol family 10 [ 3.372447] Segment Routing with IPv6 [ 3.373986] NET: Registered protocol family 17 [ 3.375905] mpls_gso: MPLS GSO support [ 3.381110] RAS: Correctable Errors collector initialized. [ 3.383403] AVX version of gcm_enc/dec engaged. [ 3.385221] AES CTR mode by8 optimization enabled [ 3.466272] sched_clock: Marking stable (3466238572, 0)->(4368518254, -902279682) [ 3.470557] registered taskstats version 1 [ 3.473195] Loading compiled-in X.509 certificates [ 3.475240] zswap: loaded using pool lzo/zbud [ 3.499337] Key type big_key registered [ 3.511260] Key type encrypted registered [ 3.512976] ima: No TPM chip found, activating TPM-bypass! [ 3.515035] ima: Allocated hash algorithm: sha1 [ 3.516994] ima: No architecture policies found [ 3.518908] evm: Initialising EVM extended attributes: [ 3.520752] evm: security.selinux [ 3.522029] evm: security.ima [ 3.523030] evm: security.capability [ 3.523991] evm: HMAC attrs: 0x1 [ 3.526156] rtc_cmos 00:05: setting system clock to 2026-08-27 18:10:29 UTC (1787854229) [ 3.532610] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 3.534806] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 3.543106] Write protecting the kernel read-only data: 28672k [ 3.546740] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.549982] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 3.589478] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.599260] systemd[1]: Detected virtualization kvm. [ 3.601203] systemd[1]: Detected architecture x86-64. [ 3.603171] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.632213] systemd[1]: No hostname configured. [ 3.634119] systemd[1]: Set hostname to . [ 3.636246] random: systemd: uninitialized urandom read (16 bytes read) [ 3.638737] systemd[1]: Initializing machine ID from random generator. [ 3.693803] random: ln: uninitialized urandom read (6 bytes read) [ 3.789730] random: systemd: uninitialized urandom read (16 bytes read) [ 3.792967] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.800579] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.806223] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.427301] device-mapper: uevent: version 1.0.3 [ 4.429563] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.153090] virtio_net virtio0 ens2: renamed from eth0 [ 5.176073] random: fast init done [ 5.229136] scsi host0: ata_piix [ 5.287275] scsi host1: ata_piix [ 5.288973] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.291468] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.796407] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.035070] random: crng init done [ 10.037932] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.458750] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.290176] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.116900] SELinux: Disabled at runtime. [ 15.235313] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.257755] systemd[1]: Detected virtualization kvm. [ 15.263378] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.818395] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.823670] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.849455] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.860628] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.869188] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.894949] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.905089] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ 17.167228] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.372179] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.134949] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.306054] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.880855] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.976561] EDAC sbridge: Ver: 1.1.2 [ 24.135797] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 24.902556] NFS: Registering the id_resolver key type [ 24.909364] Key type id_resolver registered [ 24.913157] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg315-server login: [ 71.858339] hrtimer: interrupt took 4330931 ns [ 73.715567] libcfs: loading out-of-tree module taints kernel. [ 73.788630] Key type ._llcrypt registered [ 73.791976] Key type .llcrypt registered [ 73.891939] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 90.936614] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 91.804738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.816825] alg: No test for adler32 (adler32-zlib) [ 93.103961] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 93.823204] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 95.567502] Key type lgssc registered [ 97.901432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.507762] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.418855] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 165.698506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 165.733751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.029915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.080459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.165770] Lustre: lustre-MDT0000: new disk, initializing [ 167.258422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.282584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.639199] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.432765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.521395] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.557869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.563223] Lustre: Skipped 1 previous similar message [ 182.665620] Lustre: lustre-MDT0001: new disk, initializing [ 182.726297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.749048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.757308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.452722] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.757482] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.828982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.044795] Lustre: lustre-OST0000: new disk, initializing [ 200.051217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.056743] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.135290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.366504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 206.381110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.486333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.897445] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.732399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.826236] Lustre: lustre-OST0001: new disk, initializing [ 221.829474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.835224] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.895306] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.879596] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.953491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.963202] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.011496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.923226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.746193] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.272353] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.830546] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing yml_node [ 261.798525] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 264.454959] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 266.601339] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 268.139179] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 14:14:52 EDT 2026 [ 283.684920] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 285.231687] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.632123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.645241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.678784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.656895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.666976] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.682611] Lustre: Skipped 1 previous similar message [ 307.947327] Lustre: server umount lustre-MDT0000 complete [ 311.666909] LustreError: 10385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854537 with bad export cookie 11323653046881890124 [ 311.667780] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.684298] LustreError: 10385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 311.777091] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.778981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.789421] Lustre: Skipped 2 previous similar messages [ 311.814534] Lustre: Skipped 4 previous similar messages [ 316.897640] LustreError: 6514:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.897772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.919033] LustreError: 6514:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 316.936908] Lustre: Skipped 1 previous similar message [ 318.091686] Lustre: server umount lustre-MDT0001 complete [ 322.250685] Lustre: server umount lustre-OST0000 complete [ 326.071941] Lustre: server umount lustre-OST0001 complete [ 332.535698] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 343.765597] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 377.377032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.613389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.665269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.745475] Lustre: lustre-MDT0000: new disk, initializing [ 377.818033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.831879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.723362] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 391.763986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.909480] Lustre: 15063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.982366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.984922] Lustre: Skipped 1 previous similar message [ 392.059211] Lustre: lustre-MDT0001: new disk, initializing [ 392.173471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 392.270213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 392.291348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 396.932981] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 407.272826] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 413.079800] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 420.062529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 420.425098] Lustre: lustre-OST0000: new disk, initializing [ 420.434350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 420.442895] Lustre: 17033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 420.552886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.989705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 422.009095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 422.127628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 427.341384] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 436.873719] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 441.999980] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.237853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.801609] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.066406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 453.093295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.100361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 453.115072] Lustre: Skipped 1 previous similar message [ 453.126200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.136909] LustreError: Skipped 1 previous similar message [ 453.175474] Lustre: Skipped 1 previous similar message [ 457.998418] Lustre: server umount lustre-OST0000 complete [ 463.334984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.355267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 463.368310] Lustre: Skipped 1 previous similar message [ 469.757050] Lustre: server umount lustre-MDT0000 complete [ 473.497722] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854699 with bad export cookie 11323653046881891657 [ 473.504693] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.042426] Lustre: server umount lustre-MDT0001 complete [ 478.390447] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:18:22 (1787854702) [ 485.274660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.817697] LustreError: 19298:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.839883] LustreError: 19298:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 485.944398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.504686] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 490.981465] LustreError: 19299:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.079426] LustreError: 19298:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.997991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.551635] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 511.827145] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 519.393860] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.023526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.303678] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 542.937303] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 554.490641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.490932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 554.506951] Lustre: Skipped 1 previous similar message [ 554.512365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 554.535742] Lustre: Skipped 3 previous similar messages [ 560.017713] Lustre: server umount lustre-OST0000 complete [ 569.614386] Lustre: server umount lustre-MDT0000 complete [ 569.827644] LustreError: 19298:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.848234] LustreError: 19298:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 573.252926] LustreError: 19279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854799 with bad export cookie 11323653046881892532 [ 573.257607] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.270280] LustreError: 19279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 573.789466] Lustre: server umount lustre-MDT0001 complete [ 584.938583] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 588.122642] Key type lgssc unregistered [ 588.499738] LNet: 22835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.506402] LNetError: 22835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.545856] LNet: Removed LNI 192.168.203.115@tcp [ 590.583282] Key type .llcrypt unregistered [ 590.586845] Key type ._llcrypt unregistered [ 613.948053] Key type ._llcrypt registered [ 613.950702] Key type .llcrypt registered [ 614.057123] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 615.339834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 615.467133] alg: No test for adler32 (adler32-zlib) [ 616.549656] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 616.775428] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 618.469895] Key type lgssc registered [ 619.977892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 629.239153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 629.256784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.680486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.499567] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 640.992848] LustreError: 24248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.006383] LustreError: 24248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 643.973406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.361111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 648.761302] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 656.637366] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.586189] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 671.514314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.846576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.966601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 679.336805] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 687.946342] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 704.992386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 704.996768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.006075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 705.034949] Lustre: Skipped 1 previous similar message [ 710.115064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 710.132146] Lustre: Skipped 1 previous similar message [ 715.239485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 715.247924] Lustre: Skipped 1 previous similar message [ 719.327145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 719.573333] Lustre: server umount lustre-OST0000 complete [ 725.477601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.498888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.504449] Lustre: Skipped 1 previous similar message [ 725.509074] Lustre: Skipped 1 previous similar message [ 730.011710] Lustre: server umount lustre-MDT0000 complete [ 730.597153] LustreError: 24243:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.623208] LustreError: 24243:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 733.893147] LustreError: 24229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854959 with bad export cookie 2680239593477405933 [ 733.905851] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.919804] LustreError: 24229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 734.331726] Lustre: server umount lustre-MDT0001 complete [ 746.251543] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 749.381117] Key type lgssc unregistered [ 749.757170] LNet: 28231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.775161] LNetError: 28231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.793801] LNet: Removed LNI 192.168.203.115@tcp [ 750.952411] Key type .llcrypt unregistered [ 750.959823] Key type ._llcrypt unregistered [ 778.032424] Key type ._llcrypt registered [ 778.036756] Key type .llcrypt registered [ 778.174579] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 779.232966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.293716] alg: No test for adler32 (adler32-zlib) [ 780.597760] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 781.168079] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 782.992817] Key type lgssc registered [ 784.304162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 795.587057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 795.615463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.361892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 802.727689] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 807.905935] LustreError: 29646:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.929470] LustreError: 29646:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 813.029267] LustreError: 29647:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.843141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.339645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.259623] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 829.574165] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 837.459234] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 853.463218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.843085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.311458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 867.222373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 869.100795] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 877.555869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.560044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.571750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.589745] Lustre: Skipped 1 previous similar message [ 882.657537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.673273] Lustre: Skipped 1 previous similar message [ 883.042785] Lustre: server umount lustre-OST0000 complete [ 887.777172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.800034] Lustre: Skipped 1 previous similar message [ 887.814907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.835963] Lustre: Skipped 1 previous similar message [ 892.900382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 892.909489] Lustre: Skipped 1 previous similar message [ 893.928694] Lustre: server umount lustre-MDT0000 complete [ 898.032706] LustreError: 30375:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.048637] LustreError: 30375:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 898.344961] LustreError: 31835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855124 with bad export cookie 3730508627257025531 [ 898.354785] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.361943] LustreError: 31835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 898.681455] Lustre: server umount lustre-MDT0001 complete [ 911.106600] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 914.283826] Key type lgssc unregistered [ 914.530692] LNet: 33529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.544933] LNetError: 33529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.587086] LNet: Removed LNI 192.168.203.115@tcp [ 915.581888] Key type .llcrypt unregistered [ 915.590211] Key type ._llcrypt unregistered [ 944.401079] Key type ._llcrypt registered [ 944.402829] Key type .llcrypt registered [ 944.539584] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 945.878551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.908566] alg: No test for adler32 (adler32-zlib) [ 947.088128] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 947.579956] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 949.295408] Key type lgssc registered [ 950.615448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.250744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.270401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.899847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.794393] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 970.218388] LustreError: 34944:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.248086] LustreError: 34944:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 975.331163] LustreError: 34945:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.144325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.609074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.944668] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 992.122295] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.074762] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1006.949862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.313746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.244218] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1022.648912] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1030.928331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1042.919293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1042.935896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.951921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1043.426828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1043.444605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.480852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.509450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.515498] Lustre: Skipped 1 previous similar message [ 1048.364700] Lustre: server umount lustre-OST0000 complete [ 1054.691052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1054.701710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.718446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.741606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.367466] Lustre: server umount lustre-MDT0000 complete [ 1061.858197] LustreError: 34945:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.877388] LustreError: 34945:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1062.297463] LustreError: 34925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855288 with bad export cookie 7188642879156157181 [ 1062.316567] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.674938] Lustre: server umount lustre-MDT0001 complete [ 1073.287923] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1076.088848] Key type lgssc unregistered [ 1076.303931] LNet: 38507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.318083] LNetError: 38507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.334683] LNet: Removed LNI 192.168.203.115@tcp [ 1077.153895] Key type .llcrypt unregistered [ 1077.156725] Key type ._llcrypt unregistered [ 1104.967644] Key type ._llcrypt registered [ 1104.969553] Key type .llcrypt registered [ 1105.062493] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1105.926894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.008690] alg: No test for adler32 (adler32-zlib) [ 1107.009602] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1107.361284] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1109.071318] Key type lgssc registered [ 1109.965343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1118.260075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1118.278104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.646453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.757361] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1129.957517] LustreError: 39922:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1130.010687] LustreError: 39922:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1135.083793] LustreError: 39921:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.477126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.236718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.391541] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1153.231635] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1159.596134] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1167.147656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1167.508873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.031839] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1182.265206] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1190.092370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1199.302795] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1200.611036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.612384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1200.616366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.627714] Lustre: Skipped 1 previous similar message [ 1204.395944] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1204.404722] Lustre: Skipped 1 previous similar message [ 1205.192488] Lustre: server umount lustre-OST0000 complete [ 1224.160931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1224.167534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.192771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.189348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.030441] Lustre: server umount lustre-MDT0000 complete [ 1230.306167] LustreError: 39921:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.330352] LustreError: 39921:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1233.789556] LustreError: 41766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855459 with bad export cookie 2839314428244895201 [ 1233.800912] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.223843] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1248.509740] Lustre: server umount lustre-MDT0001 complete [ 1259.497455] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1261.975414] Key type lgssc unregistered [ 1262.237305] LNet: 43656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.250979] LNetError: 43656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.265115] LNet: Removed LNI 192.168.203.115@tcp [ 1263.047698] Key type .llcrypt unregistered [ 1263.049512] Key type ._llcrypt unregistered [ 1285.915313] Key type ._llcrypt registered [ 1285.919630] Key type .llcrypt registered [ 1286.026239] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1287.217500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.249917] alg: No test for adler32 (adler32-zlib) [ 1288.340473] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1288.623744] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1290.312191] Key type lgssc registered [ 1291.308115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1300.053234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1300.072238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.623985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.374470] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1311.207829] LustreError: 45074:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.231644] LustreError: 45074:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1315.369910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.677059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.138839] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1327.126892] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1332.642158] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1339.028163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.429687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1345.694549] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1346.607401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1353.327362] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1361.106718] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1361.892567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.894625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1361.896406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.914376] Lustre: Skipped 1 previous similar message [ 1362.240460] Lustre: server umount lustre-MDT0000 complete [ 1365.650087] LustreError: 45056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855591 with bad export cookie 7000147850659395848 [ 1365.663526] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.191755] LustreError: 45071:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.221096] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 1366.233644] Lustre: Skipped 2 previous similar messages [ 1367.008202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.030879] Lustre: Skipped 1 previous similar message [ 1370.016856] LustreError: 45071:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.017273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1370.037628] LustreError: 45071:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1370.055581] Lustre: Skipped 1 previous similar message [ 1371.970856] Lustre: server umount lustre-MDT0001 complete [ 1392.563512] Lustre: server umount lustre-OST0000 complete [ 1407.853439] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1410.762183] Key type lgssc unregistered [ 1411.128989] LNet: 48938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.138098] LNetError: 48938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.159146] LNet: Removed LNI 192.168.203.115@tcp [ 1411.954235] Key type .llcrypt unregistered [ 1411.960294] Key type ._llcrypt unregistered [ 1432.134654] Key type ._llcrypt registered [ 1432.136881] Key type .llcrypt registered [ 1432.252110] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1433.290521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.311137] alg: No test for adler32 (adler32-zlib) [ 1434.328718] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1434.552468] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1436.239169] Key type lgssc registered [ 1437.066198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.037162] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.062039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1459.615511] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1464.735962] LustreError: 50302:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 1491.423903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.106499] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1506.260979] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.989236] Lustre: server umount lustre-OST0000 complete [ 1685.716327] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1687.217865] Key type lgssc unregistered [ 1687.470483] LNet: 51979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.474624] LNetError: 51979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.490435] LNet: Removed LNI 192.168.203.115@tcp [ 1688.147985] Key type .llcrypt unregistered [ 1688.151683] Key type ._llcrypt unregistered [ 1709.172463] Key type ._llcrypt registered [ 1709.174304] Key type .llcrypt registered [ 1709.334638] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1710.389889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.402944] alg: No test for adler32 (adler32-zlib) [ 1711.381507] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1711.569965] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1713.231162] Key type lgssc registered [ 1714.315857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1721.816922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1721.846110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.471686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.282156] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1733.604466] LustreError: 53394:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.619871] LustreError: 53394:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1738.727940] LustreError: 53393:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.341437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.794557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1744.416824] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1751.766528] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.202511] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1765.404741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1765.697425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.766630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1772.260543] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1779.995243] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1784.289561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.321695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.382761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.395165] Lustre: Skipped 1 previous similar message [ 1788.921785] Lustre: server umount lustre-OST0000 complete [ 1795.553769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.571385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.581605] Lustre: Skipped 1 previous similar message [ 1798.861684] Lustre: server umount lustre-MDT0000 complete [ 1800.672747] LustreError: 53393:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.697474] LustreError: 53393:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1802.606584] LustreError: 53375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856028 with bad export cookie 2830349879261197850 [ 1802.610825] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.621221] LustreError: 53375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1803.111667] Lustre: server umount lustre-MDT0001 complete [ 1814.784391] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1817.840582] Key type lgssc unregistered [ 1818.157361] LNet: 56928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.166096] LNetError: 56928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.189178] LNet: Removed LNI 192.168.203.115@tcp [ 1819.085676] Key type .llcrypt unregistered [ 1819.089254] Key type ._llcrypt unregistered [ 1840.205257] Key type ._llcrypt registered [ 1840.209246] Key type .llcrypt registered [ 1840.301810] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1841.634830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.687872] alg: No test for adler32 (adler32-zlib) [ 1842.813955] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1843.048370] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1844.727421] Key type lgssc registered [ 1845.924845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.755958] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.767598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.343881] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1874.463776] LustreError: 58313:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 1901.151228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.710444] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1916.208860] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1923.131859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.231864] LustreError: 57650:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c59de63b x1874702965410048/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1923.626947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.694533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1928.097839] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1928.676559] LustreError: 58338:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.705689] LustreError: 58338:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1933.796161] LustreError: 59371:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.858593] LustreError: 58338:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.871243] LustreError: 58338:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1936.281581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.672511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.871293] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1947.138661] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1952.425430] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.347455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.352059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.368391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.390586] Lustre: Skipped 1 previous similar message [ 1961.024463] Lustre: server umount lustre-OST0000 complete [ 1982.278907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.286666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.305226] Lustre: Skipped 1 previous similar message [ 1982.324675] Lustre: Skipped 2 previous similar messages [ 1987.553103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1987.566988] Lustre: Skipped 1 previous similar message [ 1988.645337] Lustre: server umount lustre-MDT0000 complete [ 1992.556435] LustreError: 61303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856218 with bad export cookie 10051191779328329625 [ 1992.558991] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1992.565724] LustreError: 61303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1998.971937] Lustre: server umount lustre-MDT0001 complete [ 2009.522875] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2012.072279] Key type lgssc unregistered [ 2012.283089] LNet: 62061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.291515] LNetError: 62061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.306505] LNet: Removed LNI 192.168.203.115@tcp [ 2013.148592] Key type .llcrypt unregistered [ 2013.150963] Key type ._llcrypt unregistered [ 2041.190494] Key type ._llcrypt registered [ 2041.200199] Key type .llcrypt registered [ 2041.428575] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2044.039512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.110484] alg: No test for adler32 (adler32-zlib) [ 2045.621309] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2045.896062] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2047.615650] Key type lgssc registered [ 2048.967187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2058.663475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2058.684278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.465192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.526499] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2071.013088] LustreError: 63475:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.041339] LustreError: 63475:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2072.749351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.183559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.189686] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2085.586124] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2091.072902] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2097.204953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2097.517889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.728677] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2105.735551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2111.324965] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2136.544297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.552906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.562905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2136.574698] Lustre: Skipped 1 previous similar message [ 2136.585595] Lustre: Skipped 1 previous similar message [ 2137.394408] Lustre: server umount lustre-OST0000 complete [ 2141.666137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.675557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.681349] Lustre: Skipped 1 previous similar message [ 2141.696403] Lustre: Skipped 1 previous similar message [ 2146.791490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.805032] Lustre: Skipped 1 previous similar message [ 2147.507871] Lustre: server umount lustre-MDT0000 complete [ 2150.952381] LustreError: 64212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856376 with bad export cookie 5464365185068942383 [ 2150.967235] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.250843] Lustre: server umount lustre-MDT0001 complete [ 2162.125447] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2164.660775] Key type lgssc unregistered [ 2164.980054] LNet: 67037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.991586] LNetError: 67037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.054620] LNet: Removed LNI 192.168.203.115@tcp [ 2166.986721] Key type .llcrypt unregistered [ 2166.991285] Key type ._llcrypt unregistered [ 2194.733414] Key type ._llcrypt registered [ 2194.740814] Key type .llcrypt registered [ 2194.874494] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2196.150847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2196.175779] alg: No test for adler32 (adler32-zlib) [ 2197.182898] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2197.560109] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2199.247189] Key type lgssc registered [ 2200.644795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.567756] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:47:15 (1787856435) [ 2217.728887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2217.780552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.293755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.772707] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2229.735239] LustreError: 68743:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.750720] LustreError: 68743:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2232.438081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.870230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.140198] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2244.478230] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.588935] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2258.028097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.424969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.569754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2265.281481] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2274.326239] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2281.032473] Lustre: Failing over lustre-MDT0000 [ 2281.196634] LustreError: 71322:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000df089be7) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2282.175206] LustreError: 68739:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.289901] Lustre: server umount lustre-MDT0000 complete [ 2284.002303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.008319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.015437] LustreError: 71048:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.029647] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.270088] LustreError: 68740:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.297515] LustreError: 68740:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2292.390282] LustreError: 68738:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.431517] LustreError: 68738:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2295.318790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.561261] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.860269] Lustre: *** cfs_fail_loc=135, val=0*** [ 2296.134738] LustreError: 71832:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2296.154117] LustreError: 71832:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.115@tcp: cfg command failed: rc = -2 [ 2296.172503] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2296.172503] [ 2296.183762] LustreError: MGC192.168.203.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2296.204907] LustreError: 71820:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2296.212418] LustreError: 71820:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2296.224090] LustreError: 71820:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2296.232563] LustreError: 71820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2296.262112] Lustre: server umount lustre-MDT0000 complete [ 2296.265794] LustreError: 71820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2300.394323] LustreError: 68739:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.413306] LustreError: 68739:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 2303.249692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.442553] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.449903] LustreError: 72178:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.115@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000007d505d7) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2303.465882] LustreError: 68736:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2303.653322] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (not set up) [ 2303.804119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.899908] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2308.567632] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2308.782022] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2309.114587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2309.181389] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2309.218305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2319.358816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.365178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2319.371951] Lustre: Skipped 2 previous similar messages [ 2319.386013] Lustre: Skipped 1 previous similar message [ 2322.762996] Lustre: server umount lustre-OST0000 complete [ 2329.569024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.569680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2329.600880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.065636] Lustre: server umount lustre-MDT0000 complete [ 2334.688619] LustreError: 68743:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.714122] LustreError: 68743:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2337.323439] LustreError: 68724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856563 with bad export cookie 12575541922159222472 [ 2337.332744] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.335131] LustreError: 68724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2337.771247] Lustre: server umount lustre-MDT0001 complete [ 2348.178092] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2350.274878] Key type lgssc unregistered [ 2350.554397] LNet: 73617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.561315] LNetError: 73617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.581446] LNet: Removed LNI 192.168.203.115@tcp [ 2351.473178] Key type .llcrypt unregistered [ 2351.480304] Key type ._llcrypt unregistered [ 2373.624483] Key type ._llcrypt registered [ 2373.630334] Key type .llcrypt registered [ 2373.812319] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2374.836568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.850766] alg: No test for adler32 (adler32-zlib) [ 2375.992676] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2376.411866] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2378.095176] Key type lgssc registered [ 2379.338351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2387.362743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2387.395808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.904800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.937731] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2398.187661] LustreError: 75027:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.203697] LustreError: 75027:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2403.187534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.298099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2403.600295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2408.099107] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2414.666956] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2419.973638] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2428.100601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.509681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.145269] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2434.674518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2441.739167] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2446.986174] Lustre: Failing over lustre-MDT0000 [ 2447.284483] Lustre: server umount lustre-MDT0000 complete [ 2448.590144] LustreError: 75023:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.888346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.895115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.906664] LustreError: 75027:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.404591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.429136] Lustre: Skipped 1 previous similar message [ 2453.669802] LustreError: 77469:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.684751] LustreError: 77469:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2456.005718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.088940] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.319576] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2456.490468] LustreError: 78113:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2456.501211] LustreError: 78113:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.115@tcp: cfg command failed: rc = -9 [ 2456.510615] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2456.510615] [ 2456.529456] LustreError: MGC192.168.203.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2456.540668] LustreError: 78101:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2456.555746] LustreError: 78101:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2456.571144] LustreError: 78101:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2456.582822] LustreError: 78101:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2456.614390] Lustre: server umount lustre-MDT0000 complete [ 2456.617482] LustreError: 78101:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2458.791351] LustreError: 75024:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.815334] LustreError: 75024:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2463.149251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.373571] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.640323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.689440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2463.918837] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2467.891896] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2468.860123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2468.921452] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2468.970922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2476.001707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.015528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.043949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2479.082378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.087269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2479.098429] Lustre: Skipped 1 previous similar message [ 2484.198636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.206673] Lustre: Skipped 1 previous similar message [ 2489.326072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.343186] Lustre: Skipped 1 previous similar message [ 2490.337721] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2490.611069] Lustre: server umount lustre-OST0000 complete [ 2499.556446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.556790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2499.563286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.573256] Lustre: Skipped 1 previous similar message [ 2500.797142] Lustre: server umount lustre-MDT0000 complete [ 2504.303380] LustreError: 75760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856730 with bad export cookie 11384381824319783414 [ 2504.307478] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.318875] LustreError: 75760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2504.671635] Lustre: server umount lustre-MDT0001 complete [ 2515.801942] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2518.775230] Key type lgssc unregistered [ 2519.062312] LNet: 79899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.070097] LNetError: 79899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.108085] LNet: Removed LNI 192.168.203.115@tcp [ 2521.005306] Key type .llcrypt unregistered [ 2521.011528] Key type ._llcrypt unregistered [ 2535.056599] Key type ._llcrypt registered [ 2535.060031] Key type .llcrypt registered [ 2535.160442] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2535.442319] Key type .llcrypt unregistered [ 2535.446787] Key type ._llcrypt unregistered [ 2561.934924] Key type ._llcrypt registered [ 2561.945430] Key type .llcrypt registered [ 2562.133894] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2563.168978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2563.205469] alg: No test for adler32 (adler32-zlib) [ 2564.286132] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2564.504649] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2566.167242] Key type lgssc registered [ 2567.956403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.261213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.314701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.880490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.329343] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2588.133680] LustreError: 81814:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.155228] LustreError: 81814:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2591.537535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.306883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.496130] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2603.994086] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2610.177909] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2617.667503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.159554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.292871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2624.820681] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2632.892818] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2648.036619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.063688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.074243] Lustre: Skipped 1 previous similar message [ 2653.193971] Lustre: server umount lustre-OST0000 complete [ 2658.277910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.279669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.292332] Lustre: Skipped 2 previous similar messages [ 2658.309361] Lustre: Skipped 1 previous similar message [ 2662.956587] Lustre: server umount lustre-MDT0000 complete [ 2663.395227] LustreError: 84308:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.405708] LustreError: 84308:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2666.597591] LustreError: 81794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856892 with bad export cookie 18313328021918850288 [ 2666.600429] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.607442] LustreError: 81794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2666.986424] Lustre: server umount lustre-MDT0001 complete [ 2678.715946] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2681.370221] Key type lgssc unregistered [ 2681.681904] LNet: 85374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.696841] LNetError: 85374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.727479] LNet: Removed LNI 192.168.203.115@tcp [ 2682.483472] Key type .llcrypt unregistered [ 2682.485765] Key type ._llcrypt unregistered [ 2707.892631] Key type ._llcrypt registered [ 2707.894309] Key type .llcrypt registered [ 2707.987777] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2708.828163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2708.886687] alg: No test for adler32 (adler32-zlib) [ 2710.140116] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2710.424344] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2712.153316] Key type lgssc registered [ 2713.902932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2722.938569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2722.966935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.670917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.835539] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2734.058493] LustreError: 86788:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.078854] LustreError: 86788:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2738.851249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.170180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2739.388988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.661350] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2751.825789] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2758.567942] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2766.592990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.267866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.341912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2774.698725] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2783.201942] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2789.861987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.867943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2789.873234] Lustre: Skipped 1 previous similar message [ 2789.876301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.887829] LustreError: Skipped 1 previous similar message [ 2791.389817] Lustre: server umount lustre-OST0000 complete [ 2794.978816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.980712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.997980] Lustre: Skipped 1 previous similar message [ 2795.015273] Lustre: Skipped 2 previous similar messages [ 2800.098471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.109279] Lustre: Skipped 1 previous similar message [ 2800.492447] Lustre: server umount lustre-MDT0000 complete [ 2803.855987] LustreError: 86768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857029 with bad export cookie 4988936161761708050 [ 2803.870046] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2803.875621] LustreError: 86768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2804.326099] Lustre: server umount lustre-MDT0001 complete [ 2813.317270] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2815.129644] Key type lgssc unregistered [ 2815.365497] LNet: 90321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2815.371247] LNetError: 90321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2815.380824] LNet: Removed LNI 192.168.203.115@tcp [ 2815.992592] Key type .llcrypt unregistered [ 2815.994841] Key type ._llcrypt unregistered [ 2837.867151] Key type ._llcrypt registered [ 2837.870935] Key type .llcrypt registered [ 2837.975777] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2839.143995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.165609] alg: No test for adler32 (adler32-zlib) [ 2840.322551] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2840.552793] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2842.248815] Key type lgssc registered [ 2843.504399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.768648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.793849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.352768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.783266] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2862.571729] LustreError: 91734:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.590721] LustreError: 91734:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2864.750482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.117033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.845597] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2876.972629] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.046167] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2889.284650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.632941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.446377] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2897.848210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2904.021972] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2918.380101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.381234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2918.392210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2918.414046] Lustre: Skipped 1 previous similar message [ 2923.426769] Lustre: server umount lustre-OST0000 complete [ 2928.613726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.638115] Lustre: Skipped 1 previous similar message [ 2928.663476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.667881] Lustre: Skipped 1 previous similar message [ 2933.755713] LustreError: 91735:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.771698] LustreError: 91735:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2933.837301] Lustre: server umount lustre-MDT0000 complete [ 2938.718708] LustreError: 91715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857164 with bad export cookie 13541410302383411359 [ 2938.725272] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2938.732529] LustreError: 91715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2939.204454] Lustre: server umount lustre-MDT0001 complete [ 2951.399561] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2954.288107] Key type lgssc unregistered [ 2954.659112] LNet: 95293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2954.671641] LNetError: 95293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2954.696767] LNet: Removed LNI 192.168.203.115@tcp [ 2955.685224] Key type .llcrypt unregistered [ 2955.687965] Key type ._llcrypt unregistered [ 2981.816457] Key type ._llcrypt registered [ 2981.819512] Key type .llcrypt registered [ 2981.934577] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2983.281631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2983.362282] alg: No test for adler32 (adler32-zlib) [ 2984.407937] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2984.651811] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2986.367176] Key type lgssc registered [ 2987.650273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2994.646964] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2994.672800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.271421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3015.391331] LustreError: 96685:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 3042.015217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.187350] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3056.021479] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3061.467586] Lustre: server umount lustre-OST0000 complete [ 3069.216317] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:01:34 (1787857294) [ 3075.375275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.750156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.350713] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3081.186582] LustreError: 98337:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.213344] LustreError: 98337:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3085.287319] LustreError: 98336:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.962141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.355755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.962535] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3100.710787] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3106.570199] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3113.614186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.964213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.116279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3120.194763] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3128.086231] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3140.576421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.588599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.594147] LustreError: Skipped 1 previous similar message [ 3140.596196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.618624] Lustre: Skipped 1 previous similar message [ 3140.633402] Lustre: Skipped 1 previous similar message [ 3145.704479] LustreError: 100509:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.726238] LustreError: 100509:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3145.799917] Lustre: server umount lustre-OST0000 complete [ 3150.827104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.828066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.836585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.846555] Lustre: Skipped 1 previous similar message [ 3155.353960] Lustre: server umount lustre-MDT0000 complete [ 3155.936964] LustreError: 98337:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.954584] LustreError: 98337:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3159.358711] LustreError: 98317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857385 with bad export cookie 17541796101610906393 [ 3159.369975] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.375216] LustreError: 98317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3159.630460] Lustre: server umount lustre-MDT0001 complete [ 3170.330807] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3172.853724] Key type lgssc unregistered [ 3173.097387] LNet: 102031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3173.104866] LNetError: 102031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3173.122872] LNet: Removed LNI 192.168.203.115@tcp [ 3173.827784] Key type .llcrypt unregistered [ 3173.830119] Key type ._llcrypt unregistered [ 3214.178856] Key type ._llcrypt registered [ 3214.181279] Key type .llcrypt registered [ 3214.267356] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3215.039098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.135912] alg: No test for adler32 (adler32-zlib) [ 3216.195875] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3216.440376] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3218.095175] Key type lgssc registered [ 3219.189111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.018449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.032339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.556696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.018032] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3238.893980] LustreError: 104217:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.921931] LustreError: 104217:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3241.350083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.809212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.953121] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3253.034751] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3258.591715] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3265.304414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3265.621655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.487232] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3273.837080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3278.453450] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3289.572839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.599674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.606084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.277859] Lustre: server umount lustre-OST0000 complete [ 3299.814081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.818841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.850643] Lustre: Skipped 2 previous similar messages [ 3299.879234] Lustre: Skipped 2 previous similar messages [ 3304.847416] Lustre: server umount lustre-MDT0000 complete [ 3304.930559] LustreError: 104218:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.951516] LustreError: 104218:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3308.492859] LustreError: 104198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857534 with bad export cookie 3192688218356251540 [ 3308.500979] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.505380] LustreError: 104198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3308.749137] Lustre: server umount lustre-MDT0001 complete [ 3320.387281] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3322.817698] Key type lgssc unregistered [ 3323.110788] LNet: 107753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3323.120952] LNetError: 107753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3323.135713] LNet: Removed LNI 192.168.203.115@tcp [ 3323.986925] Key type .llcrypt unregistered [ 3323.989214] Key type ._llcrypt unregistered [ 3341.645538] Key type ._llcrypt registered [ 3341.652147] Key type .llcrypt registered [ 3341.761607] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3342.964351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.008107] alg: No test for adler32 (adler32-zlib) [ 3344.063794] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3344.346729] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3346.071175] Key type lgssc registered [ 3347.184854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3355.030787] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3355.065193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.593661] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3375.713483] LustreError: 108995:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 3402.463213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.281996] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3417.517830] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.456622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.841924] LustreError: MGC192.168.203.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3424.856836] LustreError: 110018:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3424.867223] LustreError: 110018:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3424.876705] LustreError: 110018:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3424.970709] Lustre: server umount lustre-MDT0000 complete [ 3424.977138] LustreError: 110018:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3445.215165] Lustre: 108333:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857655/real 1787857655] req@0000000074be2a14 x1874704539323776/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787857671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3445.239196] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.256125] LustreError: 109007:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3445.459391] Lustre: server umount lustre-OST0000 complete [ 3456.149104] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3466.057582] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3499.964728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.210814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3500.266303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3500.337612] Lustre: lustre-MDT0000: new disk, initializing [ 3500.519687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.559521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3504.604812] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3514.327462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.483366] Lustre: 112809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.622826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3514.627533] Lustre: Skipped 1 previous similar message [ 3514.687611] Lustre: lustre-MDT0001: new disk, initializing [ 3514.913729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3514.967132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3514.983552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3519.221211] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3529.733661] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3537.597317] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3545.690429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.967183] Lustre: lustre-OST0000: new disk, initializing [ 3545.969883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3545.975372] Lustre: 114779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3546.054871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.964576] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3552.812596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3552.823217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3552.899892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3562.251202] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3566.479150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3566.672513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3570.937533] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3571.221687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3573.729339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3573.736127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.739880] LustreError: Skipped 1 previous similar message [ 3573.769496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3573.805157] Lustre: Skipped 1 previous similar message [ 3578.344220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.357576] Lustre: Skipped 1 previous similar message [ 3579.503824] Lustre: server umount lustre-OST0000 complete [ 3583.461517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.476530] Lustre: Skipped 1 previous similar message [ 3583.491608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3583.499813] Lustre: Skipped 1 previous similar message [ 3588.577990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.595231] Lustre: Skipped 1 previous similar message [ 3589.724054] Lustre: server umount lustre-MDT0000 complete [ 3593.722164] LustreError: 114778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857819 with bad export cookie 6740369878940482875 [ 3593.723781] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3593.736378] LustreError: 114778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3594.173628] Lustre: server umount lustre-MDT0001 complete [ 3602.835889] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:10:27 (1787857827) [ 3606.755375] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3624.261930] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3633.133305] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3669.627381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.946404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3669.985533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3670.056770] Lustre: lustre-MDT0000: new disk, initializing [ 3670.125747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.143240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3674.743620] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3684.527576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.631897] Lustre: 119680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3684.754949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3684.759693] Lustre: Skipped 1 previous similar message [ 3684.812144] Lustre: lustre-MDT0001: new disk, initializing [ 3684.876613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3684.920056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3684.934228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3689.485203] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3699.964501] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3705.512547] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3711.757101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.961085] Lustre: lustre-OST0000: new disk, initializing [ 3711.965484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3711.974926] Lustre: 121648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3712.045317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.304460] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3720.793206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3720.810517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3720.939403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3730.578925] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3736.609424] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3736.789885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3741.522917] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3741.840358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3746.786411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3746.792619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.810864] Lustre: Skipped 1 previous similar message [ 3746.820403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.830713] Lustre: Skipped 1 previous similar message [ 3750.397783] Lustre: server umount lustre-OST0000 complete [ 3756.512690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3756.523365] LustreError: Skipped 1 previous similar message [ 3756.534381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.553844] Lustre: Skipped 1 previous similar message [ 3756.562050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3760.540810] Lustre: server umount lustre-MDT0000 complete [ 3762.147775] LustreError: 119692:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.160917] LustreError: 119692:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 3765.260927] LustreError: 120475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857991 with bad export cookie 6740369878940483757 [ 3765.262707] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.269527] LustreError: 120475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3765.632300] Lustre: server umount lustre-MDT0001 complete [ 3773.695651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.255626] LustreError: 123864:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3774.357763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3778.969265] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3779.566310] LustreError: 123865:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3784.682366] LustreError: 123864:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.157360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.883696] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3801.789107] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3807.902635] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3815.572341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.321507] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3832.451383] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3840.752168] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3848.673547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.709482] Lustre: Skipped 1 previous similar message [ 3848.736204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3848.746678] Lustre: Skipped 3 previous similar messages [ 3852.435973] Lustre: server umount lustre-OST0000 complete [ 3858.913901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.935638] Lustre: Skipped 1 previous similar message [ 3862.771528] Lustre: server umount lustre-MDT0000 complete [ 3864.033543] LustreError: 123860:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.048752] LustreError: 123860:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3866.782658] LustreError: 123844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858092 with bad export cookie 6740369878940484632 [ 3866.788580] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3866.802479] LustreError: 123844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3867.140639] Lustre: server umount lustre-MDT0001 complete [ 3878.064238] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3880.864305] Key type lgssc unregistered [ 3881.153880] LNet: 127503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3881.165558] LNetError: 127503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3881.181464] LNet: Removed LNI 192.168.203.115@tcp [ 3882.007837] Key type .llcrypt unregistered [ 3882.010943] Key type ._llcrypt unregistered [ 3899.070725] Key type ._llcrypt registered [ 3899.072958] Key type .llcrypt registered [ 3899.190784] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3916.091747] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3917.178914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3917.238656] alg: No test for adler32 (adler32-zlib) [ 3918.536032] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3918.927471] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3920.727170] Key type lgssc registered [ 3922.418834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3963.186870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3963.259703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.598646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3964.654273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3964.760502] Lustre: lustre-MDT0000: new disk, initializing [ 3964.835393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.855219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3969.247755] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3979.574732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.779212] Lustre: 130660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3979.840554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3979.847405] Lustre: Skipped 1 previous similar message [ 3979.959138] Lustre: lustre-MDT0001: new disk, initializing [ 3980.122904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.175060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3980.190715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3985.359859] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3995.009780] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4001.029722] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4007.229920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4007.505460] Lustre: lustre-OST0000: new disk, initializing [ 4007.518249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4007.530412] Lustre: 132630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4007.643407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.805125] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4015.686151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4015.691729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4015.787547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4022.722309] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4027.799077] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4028.093116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4033.959603] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4034.263496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4036.583479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4036.584863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.596059] LustreError: Skipped 1 previous similar message [ 4036.597074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4036.620711] Lustre: Skipped 1 previous similar message [ 4041.704163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4041.709430] Lustre: Skipped 2 previous similar messages [ 4042.729305] Lustre: server umount lustre-OST0000 complete [ 4046.820446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4046.823356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.833736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.868653] Lustre: Skipped 1 previous similar message [ 4051.939299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.951697] Lustre: Skipped 2 previous similar messages [ 4052.410974] Lustre: server umount lustre-MDT0000 complete [ 4055.726573] LustreError: 130654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858281 with bad export cookie 12890314719277348000 [ 4055.727387] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4055.739607] LustreError: 130654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4055.972098] Lustre: server umount lustre-MDT0001 complete [ 4063.657366] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:18:08 (1787858288) [ 4069.181954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.618316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.504511] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4074.983459] LustreError: 135043:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.016515] LustreError: 135043:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4079.076084] LustreError: 135044:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.825104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.181508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.034374] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4092.733985] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4098.229200] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4102.625180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4102.643457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.667472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.686827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.866378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4112.876049] Lustre: Skipped 5 previous similar messages [ 4114.399590] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4114.534157] Lustre: server umount lustre-MDT0000 complete [ 4117.948314] LustreError: 135023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858343 with bad export cookie 12890314719277348882 [ 4117.967129] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4118.344871] Lustre: server umount lustre-MDT0001 complete [ 4126.640453] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:19:11 (1787858351) [ 4133.498520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4149.152707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4154.271470] LustreError: 137460:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 4179.999248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.150537] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4193.303755] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4195.623645] Lustre: server umount lustre-OST0000 complete [ 4204.083890] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:20:28 (1787858428) [ 4210.568459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4211.089292] LustreError: 138906:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.162457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.285264] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4216.290848] LustreError: 138907:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.419632] LustreError: 138906:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4223.272442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.887530] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4235.560775] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4242.376949] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4249.556610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.914400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4249.920382] Lustre: Skipped 1 previous similar message [ 4256.837517] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4265.575589] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4280.290534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4280.294709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.305040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4285.163352] Lustre: server umount lustre-MDT0000 complete [ 4288.335675] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858514 with bad export cookie 12890314719277349540 [ 4288.344671] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4288.345608] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4288.636323] Lustre: server umount lustre-MDT0001 complete [ 4302.050683] Lustre: server umount lustre-OST0000 complete [ 4309.438811] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:22:14 (1787858534) [ 4315.818815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.140632] LustreError: 142584:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.224758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.252615] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4327.889894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.400288] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4339.865821] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4346.314236] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4353.134898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.410574] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4361.720496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4368.328575] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.799732] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4374.170231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4379.443887] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4379.713728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4382.184397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.192614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4382.193122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4382.193128] Lustre: Skipped 3 previous similar messages [ 4382.209627] Lustre: Skipped 3 previous similar messages [ 4396.002151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4396.243775] Lustre: server umount lustre-OST0000 complete [ 4400.098404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4400.109642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.016334] Lustre: server umount lustre-MDT0000 complete [ 4407.780759] LustreError: 142579:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.807737] LustreError: 142579:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4410.366614] LustreError: 142565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858636 with bad export cookie 12890314719277350618 [ 4410.369778] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4410.384184] LustreError: 142565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.726378] Lustre: server umount lustre-MDT0001 complete [ 4421.713461] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:24:05 (1787858645) [ 4428.686598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.319590] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4449.439458] LustreError: 146745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 4475.039730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.054887] Lustre: Skipped 2 previous similar messages [ 4482.961080] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4491.679882] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4499.124296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.683710] LustreError: 146771:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.820612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4504.327787] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4513.634808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.230759] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4527.051794] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4532.768828] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4536.908142] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4537.128308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4541.439140] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4541.669930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4545.005413] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.010910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4545.016965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4545.016974] Lustre: Skipped 8 previous similar messages [ 4545.018576] Lustre: Skipped 2 previous similar messages [ 4558.303149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4558.546937] Lustre: server umount lustre-MDT0000 complete [ 4562.001595] LustreError: 146751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858787 with bad export cookie 12890314719277351640 [ 4562.011234] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.387752] Lustre: server umount lustre-MDT0001 complete [ 4576.148825] Lustre: server umount lustre-OST0000 complete [ 4584.405194] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:26:49 (1787858809) [ 4590.368601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4605.919884] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4611.103390] LustreError: 150940:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 4642.523966] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4649.553465] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4655.562475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.860527] LustreError: 150964:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.869800] LustreError: 150964:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 4655.962452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4660.364850] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4668.378513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.184081] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4680.693335] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4686.740371] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4693.593311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.718957] Lustre: lustre-OST0001: new disk, initializing [ 4693.721840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4693.728783] Lustre: 153810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4699.461077] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4703.827406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4703.844347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4704.017926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4710.571475] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4715.998984] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4716.254324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4720.635314] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4720.915630] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4724.710621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.719341] Lustre: Skipped 1 previous similar message [ 4724.722879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.726626] Lustre: Skipped 9 previous similar messages [ 4736.991320] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4737.116658] Lustre: server umount lustre-OST0000 complete [ 4745.197410] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4746.807521] Lustre: server umount lustre-OST0001 complete [ 4756.704623] Lustre: server umount lustre-MDT0000 complete [ 4759.800297] LustreError: 153809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858985 with bad export cookie 12890314719277352592 [ 4759.813654] LustreError: 153809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4759.819875] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4760.156691] Lustre: server umount lustre-MDT0001 complete [ 4781.637589] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:30:06 (1787859006) [ 4783.020261] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4784.985629] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4786.515543] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:30:11 (1787859011) [ 4792.637531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.955366] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4793.003231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4793.352507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4793.364768] Lustre: Skipped 6 previous similar messages [ 4798.329143] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4808.428612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.529466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4808.548414] Lustre: 157372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4813.437880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4820.587626] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4825.512898] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4831.215971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.328016] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4836.972493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4837.816206] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4845.413799] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.285988] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4850.510291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4855.685337] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4855.894616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4857.827863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4857.832353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.832984] LustreError: Skipped 2 previous similar messages [ 4857.842975] Lustre: Skipped 5 previous similar messages [ 4872.159444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4872.324993] Lustre: server umount lustre-OST0000 complete [ 4894.931521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4898.339720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.356766] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4901.489240] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4909.182331] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4915.250506] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4915.557531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4918.819067] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.839582] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4920.703795] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4920.903917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4926.183994] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e1e600db000.ost_server_uuid 50 [ 4927.654759] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e1e600db000.ost_server_uuid in FULL state after 0 sec [ 4927.827836] LustreError: 157377:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.15@tcp) returned error from blocking AST (req@00000000908fafd6 x1874705141472128 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000048482fba/0xb2e398f0fd840fea lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.15@tcp remote: 0xb8f4813d80e03fb2 expref: 5 pid: 157378 timeout: 5027 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4927.871722] LustreError: lustre-MDT0000: A client on nid 192.168.203.15@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4927.879183] LustreError: 157367:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.15@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000048482fba/0xb2e398f0fd840fea lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.15@tcp remote: 0xb8f4813d80e03fb2 expref: 6 pid: 157378 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4937.802116] Lustre: server umount lustre-OST0000 complete [ 4947.934863] Lustre: server umount lustre-MDT0000 complete [ 4949.475705] LustreError: 157382:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.490573] LustreError: 157382:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 4951.201229] LustreError: 157363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859177 with bad export cookie 12890314719277353775 [ 4951.216853] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.217200] LustreError: 157363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4957.596961] Lustre: server umount lustre-MDT0001 complete [ 4968.459460] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4971.034669] Key type lgssc unregistered [ 4971.288849] LNet: 163497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4971.302306] LNetError: 163497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4971.324874] LNet: Removed LNI 192.168.203.115@tcp [ 4972.172183] Key type .llcrypt unregistered [ 4972.176342] Key type ._llcrypt unregistered [ 4994.107540] Key type ._llcrypt registered [ 4994.110810] Key type .llcrypt registered [ 4994.240409] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4995.471633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4995.540287] alg: No test for adler32 (adler32-zlib) [ 4996.552831] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4996.847610] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4998.495724] Key type lgssc registered [ 4999.639409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5008.352017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5008.395249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.920853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5014.473609] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5019.104744] LustreError: 164912:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.113585] LustreError: 164912:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5022.521681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.866054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5027.056823] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5033.626229] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5038.582151] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5044.686431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5045.065536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.849750] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5054.262391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5057.377687] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5063.772426] Lustre: Failing over lustre-MDT0000 [ 5063.989088] Lustre: server umount lustre-MDT0000 complete [ 5064.674872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.684748] LustreError: 165639:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.687244] Lustre: Skipped 2 previous similar messages [ 5064.727252] LustreError: 165639:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5069.800031] LustreError: 164913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.815540] LustreError: 164913:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5074.913263] LustreError: 164912:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.925232] LustreError: 164912:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5075.702249] LustreError: 164892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.15@tcp arrived at 1787859301 with bad export cookie 6625482828941787222 [ 5075.708969] LustreError: 164892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5080.032542] LustreError: 165639:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.047649] LustreError: 165639:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5081.055305] Lustre: 164221:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859290/real 1787859290] req@0000000073b0b248 x1874706271593600/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787859306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5081.087695] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5081.098759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.296710] LustreError: 164913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.300788] LustreError: 164219:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f7a6adf8 x1874706271599744/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5091.328030] LustreError: 164913:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 5091.630689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.730287] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5095.461556] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5096.931973] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5096.938847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5097.242754] Lustre: lustre-MDT0000: Denying connection for new client e7bc82d7-e537-4034-823d-69cfc0293397 (at 192.168.203.15@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5107.168780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.181782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5107.183133] Lustre: Skipped 1 previous similar message [ 5111.578924] Lustre: server umount lustre-OST0000 complete [ 5115.077381] LustreError: 168642:0:(ldlm_lib.c:3039:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5115.082153] Lustre: 167991:0:(ldlm_lib.c:2439:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5115.090531] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5115.106253] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5115.116833] LustreError: 167991:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f03feead x1874706271613312/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5115.132232] LustreError: 167991:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5115.137436] LustreError: 167991:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5115.145788] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5115.436460] Lustre: server umount lustre-MDT0000 complete [ 5117.411905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.427482] LustreError: 164909:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.440483] LustreError: 167489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859344 with bad export cookie 6625482828941787320 [ 5118.444198] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.449868] LustreError: 167489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5118.463584] LustreError: 164219:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5124.762806] Lustre: server umount lustre-MDT0001 complete [ 5134.438900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5137.824458] Key type lgssc unregistered [ 5138.201816] LNet: 169397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5138.218826] LNetError: 169397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5138.235180] LNet: Removed LNI 192.168.203.115@tcp [ 5139.252268] Key type .llcrypt unregistered [ 5139.254293] Key type ._llcrypt unregistered [ 5159.645726] Key type ._llcrypt registered [ 5159.647035] Key type .llcrypt registered [ 5159.734669] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5160.603494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5160.644204] alg: No test for adler32 (adler32-zlib) [ 5161.627506] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5161.840889] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5163.503206] Key type lgssc registered [ 5164.224236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5170.480671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5170.507458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.890938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.487743] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5181.411385] LustreError: 170791:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.435295] LustreError: 170791:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5182.922169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.186722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5187.152802] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5193.628787] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.423646] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.899192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.403825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.137586] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5216.697896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5218.622433] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5226.989749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5226.995700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.996549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5227.030844] Lustre: Skipped 1 previous similar message [ 5231.257805] Lustre: server umount lustre-OST0000 complete [ 5237.219400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.222712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5237.237048] Lustre: Skipped 1 previous similar message [ 5237.251434] Lustre: Skipped 2 previous similar messages [ 5241.304060] Lustre: server umount lustre-MDT0000 complete [ 5242.342036] LustreError: 170790:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.356728] LustreError: 170790:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5244.928970] LustreError: 170772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859470 with bad export cookie 9857997637473627479 [ 5244.945711] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.945872] LustreError: 170772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5245.373957] Lustre: server umount lustre-MDT0001 complete [ 5256.175888] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5258.581526] Key type lgssc unregistered [ 5258.857915] LNet: 174326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5258.868283] LNetError: 174326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5258.883399] LNet: Removed LNI 192.168.203.115@tcp [ 5259.614219] Key type .llcrypt unregistered [ 5259.618127] Key type ._llcrypt unregistered [ 5288.053528] Key type ._llcrypt registered [ 5288.057513] Key type .llcrypt registered [ 5288.166252] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5289.033728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5289.148233] alg: No test for adler32 (adler32-zlib) [ 5290.265904] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5290.507817] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5292.175267] Key type lgssc registered [ 5293.224065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5300.587814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5300.609632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.958507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.920667] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5311.467534] LustreError: 176024:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.496084] LustreError: 176024:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5314.560091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.058184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5319.384331] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5326.460832] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5331.050501] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5336.679238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5337.025085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.612343] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5346.234152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5349.691270] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5356.514178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5356.520075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.538134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5356.549979] Lustre: Skipped 1 previous similar message [ 5361.645774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.654468] Lustre: Skipped 1 previous similar message [ 5362.181105] Lustre: server umount lustre-OST0000 complete [ 5366.248092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5366.256563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.278426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.285660] Lustre: Skipped 1 previous similar message [ 5371.875095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.879632] Lustre: Skipped 1 previous similar message [ 5372.527577] Lustre: server umount lustre-MDT0000 complete [ 5376.995926] LustreError: 176025:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.027650] LustreError: 176025:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5377.266103] LustreError: 177866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859603 with bad export cookie 13218653680566137924 [ 5377.271193] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5377.281640] LustreError: 177866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5377.785256] Lustre: server umount lustre-MDT0001 complete [ 5389.745237] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5392.532575] Key type lgssc unregistered [ 5392.769517] LNet: 179559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5392.774341] LNetError: 179559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5392.810512] LNet: Removed LNI 192.168.203.115@tcp [ 5393.486090] Key type .llcrypt unregistered [ 5393.489531] Key type ._llcrypt unregistered [ 5417.445784] Key type ._llcrypt registered [ 5417.448339] Key type .llcrypt registered [ 5417.587643] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5418.562301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5418.617397] alg: No test for adler32 (adler32-zlib) [ 5419.879072] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5420.093728] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5421.759191] Key type lgssc registered [ 5422.934712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5434.284407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5434.304773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.842754] Lustre: *** cfs_fail_loc=135, val=0*** [ 5435.945605] LustreError: 181084:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5435.959094] LustreError: 181084:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.115@tcp: cfg command failed: rc = -2 [ 5435.975953] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5435.975953] [ 5435.991119] LustreError: MGC192.168.203.115@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5436.002279] LustreError: 181045:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5436.011940] LustreError: 181045:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5436.025303] LustreError: 181045:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5436.034540] LustreError: 181045:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5436.122100] Lustre: server umount lustre-MDT0000 complete [ 5436.130639] LustreError: 181045:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5442.141572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.453501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5446.448676] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5452.771894] LustreError: 181455:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.812069] LustreError: 181455:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5454.730969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.051033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5459.619488] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5466.358629] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5471.695085] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5475.812277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5475.828412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.856631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5477.861478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5477.870422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.804432] Lustre: server umount lustre-MDT0000 complete [ 5479.969739] LustreError: 181450:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.013689] LustreError: 181450:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5483.399098] LustreError: 181435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859709 with bad export cookie 7041596750507032730 [ 5483.406322] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5483.407289] LustreError: 181435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5483.794725] Lustre: server umount lustre-MDT0001 complete [ 5493.348491] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5495.836671] Key type lgssc unregistered [ 5496.196659] LNet: 183781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5496.210771] LNetError: 183781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5496.236679] LNet: Removed LNI 192.168.203.115@tcp [ 5497.041716] Key type .llcrypt unregistered [ 5497.043355] Key type ._llcrypt unregistered [ 5518.189854] Key type ._llcrypt registered [ 5518.194798] Key type .llcrypt registered [ 5518.317733] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5518.576131] Key type .llcrypt unregistered [ 5518.579799] Key type ._llcrypt unregistered [ 5533.446759] Key type ._llcrypt registered [ 5533.450108] Key type .llcrypt registered [ 5533.589210] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5534.510451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5534.596160] alg: No test for adler32 (adler32-zlib) [ 5535.603876] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5535.849271] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5537.513350] Key type lgssc registered [ 5538.573870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5546.297759] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5546.323834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.015681] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5567.135722] LustreError: 185898:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 5593.823364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.457689] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5607.708337] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5614.001227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.543949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.619786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5618.724532] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5619.695852] LustreError: 185924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.742899] LustreError: 185924:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5624.805759] LustreError: 185923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.990940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5627.516296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5632.098641] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5639.786860] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5645.339473] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5647.848634] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5658.080553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5658.096597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.114946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5659.619828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5659.635530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.170387] Lustre: server umount lustre-OST0000 complete [ 5669.867439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.884390] Lustre: Skipped 1 previous similar message [ 5669.887484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.894682] Lustre: Skipped 1 previous similar message [ 5674.989582] LustreError: 186941:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.022362] LustreError: 186941:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5675.133455] Lustre: server umount lustre-MDT0000 complete [ 5679.988447] LustreError: 185906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859905 with bad export cookie 8873721406354511085 [ 5679.996362] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5680.295930] Lustre: server umount lustre-MDT0001 complete [ 5692.542591] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5695.510430] Key type lgssc unregistered [ 5695.818801] LNet: 189868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5695.838702] LNetError: 189868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5695.858314] LNet: Removed LNI 192.168.203.115@tcp [ 5696.657956] Key type .llcrypt unregistered [ 5696.660751] Key type ._llcrypt unregistered [ 5719.958213] Key type ._llcrypt registered [ 5719.963037] Key type .llcrypt registered [ 5720.066773] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5721.152804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5721.181495] alg: No test for adler32 (adler32-zlib) [ 5722.174140] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5722.366334] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5724.055144] Key type lgssc registered [ 5724.900537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5731.209368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5731.226197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5732.632579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.065900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5742.052061] LustreError: 191279:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.065793] LustreError: 191279:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5742.779681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.245443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.309919] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5755.012657] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5760.371496] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5766.948818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5767.280762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.180702] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5777.537963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5780.879332] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5786.496293] Lustre: Failing over lustre-MDT0000 [ 5786.783542] Lustre: server umount lustre-MDT0000 complete [ 5788.134172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.141599] LustreError: 191276:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.154227] Lustre: Skipped 2 previous similar messages [ 5788.172934] LustreError: 191276:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5793.248939] LustreError: 191279:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.260831] LustreError: 191279:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5798.373046] LustreError: 191279:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.388912] LustreError: 191279:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5803.495064] LustreError: 192343:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5803.517920] LustreError: 192343:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5804.512669] Lustre: 190592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860014/real 1787860014] req@00000000843cd13d x1874707032858752/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787860030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5804.550141] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5804.716480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.763644] LustreError: 191276:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.763924] LustreError: 190591:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa5775ed x1874707032864896/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5814.787767] LustreError: 191276:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 5815.085193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.172508] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5815.458601] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5819.037454] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5820.417081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5820.434536] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5820.490066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5823.888781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5830.312522] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5840.357058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5840.372531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.388671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5844.109633] Lustre: server umount lustre-OST0000 complete [ 5851.104972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.106759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.112645] Lustre: Skipped 2 previous similar messages [ 5851.134440] Lustre: Skipped 2 previous similar messages [ 5853.617245] Lustre: server umount lustre-MDT0000 complete [ 5856.224430] LustreError: 195082:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.232329] LustreError: 195082:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5856.403365] LustreError: 194861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860082 with bad export cookie 8914583793323311057 [ 5856.407173] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5856.417292] LustreError: 194861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5856.678636] Lustre: server umount lustre-MDT0001 complete [ 5865.993062] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5868.729285] Key type lgssc unregistered [ 5869.062120] LNet: 196125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5869.067177] LNetError: 196125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5869.095769] LNet: Removed LNI 192.168.203.115@tcp [ 5870.329077] Key type .llcrypt unregistered [ 5870.331147] Key type ._llcrypt unregistered [ 5895.903338] Key type ._llcrypt registered [ 5895.905466] Key type .llcrypt registered [ 5896.028255] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5897.601811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5897.647819] alg: No test for adler32 (adler32-zlib) [ 5898.674808] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5898.898789] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5900.632836] Key type lgssc registered [ 5902.364606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5909.895603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5909.910826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.379272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.739971] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5921.770765] LustreError: 197536:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.794761] LustreError: 197536:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5924.145606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.510457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5928.485674] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5934.883832] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5940.186362] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5946.610461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.863618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.470571] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5956.153124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5960.941433] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5964.667652] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5968.057071] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5982.131170] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5990.882036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5990.889772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.914339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5991.912832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5991.924091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5991.928270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.967361] Lustre: Skipped 1 previous similar message [ 5995.756581] Lustre: server umount lustre-OST0000 complete [ 6001.120439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6001.134777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.163248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.621300] Lustre: server umount lustre-MDT0000 complete [ 6007.265155] LustreError: 198264:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.281611] LustreError: 198264:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6009.129065] LustreError: 197519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860235 with bad export cookie 16492618237397365414 [ 6009.133703] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6009.143709] LustreError: 197519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6009.475916] Lustre: server umount lustre-MDT0001 complete [ 6021.088918] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6023.969916] Key type lgssc unregistered [ 6024.193178] LNet: 201386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.197650] LNetError: 201386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.212476] LNet: Removed LNI 192.168.203.115@tcp [ 6025.113248] Key type .llcrypt unregistered [ 6025.116505] Key type ._llcrypt unregistered [ 6044.139677] Key type ._llcrypt registered [ 6044.142818] Key type .llcrypt registered [ 6044.251604] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6045.244410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6045.323253] alg: No test for adler32 (adler32-zlib) [ 6046.342816] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6046.583585] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6048.311197] Key type lgssc registered [ 6049.566248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6056.323621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6056.341600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.755204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.657325] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6067.170823] LustreError: 202652:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.186636] LustreError: 202652:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6071.411245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.895297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.300507] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6083.340969] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6089.168201] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6097.620162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.231615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.364987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6105.357071] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6114.807408] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6120.512380] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6133.954937] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6144.524515] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6153.991959] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6164.965085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.969353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.984250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.036432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6168.040589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.052855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.083353] Lustre: Skipped 1 previous similar message [ 6170.592270] Lustre: server umount lustre-OST0000 complete [ 6178.273515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.300695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.317195] Lustre: Skipped 1 previous similar message [ 6181.318795] Lustre: server umount lustre-MDT0000 complete [ 6183.393097] LustreError: 203381:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.413938] LustreError: 203381:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6184.977714] LustreError: 202632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860410 with bad export cookie 8179134119885845768 [ 6184.984188] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6184.991663] LustreError: 202632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6185.388491] Lustre: server umount lustre-MDT0001 complete [ 6197.098901] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6200.260814] Key type lgssc unregistered [ 6200.627281] LNet: 207775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6200.643189] LNetError: 207775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6200.679566] LNet: Removed LNI 192.168.203.115@tcp [ 6201.607734] Key type .llcrypt unregistered [ 6201.609304] Key type ._llcrypt unregistered [ 6226.328675] Key type ._llcrypt registered [ 6226.332173] Key type .llcrypt registered [ 6226.421627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6227.375721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6227.430536] alg: No test for adler32 (adler32-zlib) [ 6228.496240] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6228.748330] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6230.439167] Key type lgssc registered [ 6231.554807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6239.010311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6239.032383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.543563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.985406] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6250.980956] LustreError: 209191:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6251.001854] LustreError: 209191:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6253.908910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.135847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.397134] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6266.037696] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6271.909390] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6278.291656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.567808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.506951] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6285.745570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6291.149689] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6306.272701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6306.278774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.287702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6306.292979] Lustre: Skipped 1 previous similar message [ 6306.309644] Lustre: Skipped 1 previous similar message [ 6311.396158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6311.400574] Lustre: Skipped 1 previous similar message [ 6312.228653] Lustre: server umount lustre-OST0000 complete [ 6316.513327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.518609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6316.543517] Lustre: Skipped 1 previous similar message [ 6316.555964] Lustre: Skipped 1 previous similar message [ 6321.638320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6321.643822] Lustre: Skipped 1 previous similar message [ 6322.746044] Lustre: server umount lustre-MDT0000 complete [ 6326.464274] LustreError: 209170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860552 with bad export cookie 942387413584564191 [ 6326.468250] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6326.479707] LustreError: 209170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6326.848569] Lustre: server umount lustre-MDT0001 complete [ 6338.466453] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6341.600821] Key type lgssc unregistered [ 6342.035537] LNet: 212877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6342.047294] LNetError: 212877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6342.063867] LNet: Removed LNI 192.168.203.115@tcp [ 6343.141760] Key type .llcrypt unregistered [ 6343.144396] Key type ._llcrypt unregistered [ 6362.133593] Key type ._llcrypt registered [ 6362.142365] Key type .llcrypt registered [ 6362.276141] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6363.212439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6363.259328] alg: No test for adler32 (adler32-zlib) [ 6364.339307] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6364.591623] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6366.247424] Key type lgssc registered [ 6367.270883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6374.588572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6374.612431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.172198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.842675] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6386.663642] LustreError: 214141:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.684700] LustreError: 214141:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6389.293492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.631625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.805101] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6401.018631] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6406.004248] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6412.126752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6412.511970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.327658] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6421.744283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6425.808731] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6440.422120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.434953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.459841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6442.464322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.467417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6442.476226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.433847] Lustre: server umount lustre-OST0000 complete [ 6451.169494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6451.177689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.194533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6451.200550] Lustre: Skipped 1 previous similar message [ 6455.389711] Lustre: server umount lustre-MDT0000 complete [ 6457.858679] LustreError: 214141:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.901091] LustreError: 214141:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6459.057761] LustreError: 214122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860685 with bad export cookie 4116432370624368105 [ 6459.064557] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.070430] LustreError: 214122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6459.489633] Lustre: server umount lustre-MDT0001 complete [ 6471.168468] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6474.888910] Key type lgssc unregistered [ 6475.247899] LNet: 217826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6475.253334] LNetError: 217826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6475.298606] LNet: Removed LNI 192.168.203.115@tcp [ 6476.410983] Key type .llcrypt unregistered [ 6476.415809] Key type ._llcrypt unregistered [ 6494.616752] Key type ._llcrypt registered [ 6494.619413] Key type .llcrypt registered [ 6494.773655] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6496.133334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6496.148623] alg: No test for adler32 (adler32-zlib) [ 6497.209623] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6497.459962] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6499.152116] Key type lgssc registered [ 6500.055428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6506.597686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6506.613146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.025533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.696744] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6517.222262] LustreError: 219092:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.236318] LustreError: 219092:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6518.863945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.175371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6522.875177] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6529.415278] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6534.338936] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6541.621353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6541.986562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.586960] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6552.305015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6554.748924] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.050642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6564.104938] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6564.334307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6571.304266] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6579.362667] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6581.177700] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6592.994304] Lustre: Permanently deactivating lustre-OST0001 [ 6593.012240] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6598.245853] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6606.658492] Lustre: Permanently reactivating lustre-OST0001 [ 6606.680365] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6606.684822] Lustre: Skipped 2 previous similar messages [ 6606.795186] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.815583] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6606.830440] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6606.857240] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6606.920388] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6622.177589] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6622.190470] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.218791] Lustre: Skipped 1 previous similar message [ 6622.233898] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6622.688363] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.249520] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6625.263878] Lustre: Skipped 1 previous similar message [ 6628.405076] Lustre: server umount lustre-OST0001 complete [ 6630.388336] LustreError: 220912:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.412419] LustreError: 220912:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6634.469424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6634.475726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.486964] Lustre: Skipped 1 previous similar message [ 6634.492586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6635.497315] LustreError: 223007:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.528647] LustreError: 223007:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6639.437829] Lustre: server umount lustre-OST0000 complete [ 6645.732209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.735416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6645.749484] Lustre: Skipped 2 previous similar messages [ 6645.765769] Lustre: Skipped 3 previous similar messages [ 6648.984988] Lustre: server umount lustre-MDT0000 complete [ 6650.848357] LustreError: 219092:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6650.880925] LustreError: 219092:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6652.587845] LustreError: 219071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860878 with bad export cookie 8163673797601611548 [ 6652.596572] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.607588] LustreError: 219071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6652.893795] Lustre: server umount lustre-MDT0001 complete [ 6663.669674] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6666.323910] Key type lgssc unregistered [ 6666.605829] LNet: 224176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6666.612073] LNetError: 224176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6666.624319] LNet: Removed LNI 192.168.203.115@tcp [ 6667.434326] Key type .llcrypt unregistered [ 6667.440185] Key type ._llcrypt unregistered [ 6706.524967] Key type ._llcrypt registered [ 6706.530242] Key type .llcrypt registered [ 6706.643550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6707.996970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6708.020597] alg: No test for adler32 (adler32-zlib) [ 6709.025907] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6709.248435] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6710.911200] Key type lgssc registered [ 6711.972414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6718.826420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6718.838819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.087742] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6720.120995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6720.289443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6725.208485] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6736.424499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.645489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6736.715393] Lustre: 226381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6737.221302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6744.227977] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6753.492273] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.461309] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.009942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.170768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6767.314493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.481450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6774.348257] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6782.318850] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6786.362795] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6792.604484] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6800.540877] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6806.580753] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6816.041320] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6834.616326] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6834.623891] Lustre: Skipped 1 previous similar message [ 6871.470644] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6871.481774] Lustre: Skipped 3 previous similar messages [ 6940.099222] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6940.114943] Lustre: Skipped 9 previous similar messages [ 6957.536289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6957.542077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.567530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6958.065460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6958.076055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.095509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6958.103119] Lustre: Skipped 1 previous similar message [ 6963.173538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6963.185111] Lustre: Skipped 1 previous similar message [ 6963.595707] Lustre: server umount lustre-OST0000 complete [ 6967.793920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6967.803349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.838996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6973.614673] Lustre: server umount lustre-MDT0000 complete [ 6977.727811] LustreError: 226372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861203 with bad export cookie 6843782118443627430 [ 6977.736309] LustreError: 226372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6977.738348] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6978.101057] Lustre: server umount lustre-MDT0001 complete [ 6989.745910] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6992.753688] Key type lgssc unregistered [ 6993.066201] LNet: 231178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6993.076594] LNetError: 231178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6993.109303] LNet: Removed LNI 192.168.203.115@tcp [ 6993.939163] Key type .llcrypt unregistered [ 6993.941871] Key type ._llcrypt unregistered [ 7018.363835] Key type ._llcrypt registered [ 7018.365695] Key type .llcrypt registered [ 7018.463835] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7019.535650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7019.568036] alg: No test for adler32 (adler32-zlib) [ 7020.737419] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7020.962530] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7022.671432] Key type lgssc registered [ 7023.569735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7030.114658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7030.142231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.509219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.671845] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7042.016819] LustreError: 232593:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.035094] LustreError: 232593:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7043.150465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.476956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7047.443665] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7053.976749] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7059.138791] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.619301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.917860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.728246] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7076.159507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7078.392636] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.817338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.827934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.848846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.854596] Lustre: Skipped 1 previous similar message [ 7097.889309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.901416] Lustre: Skipped 1 previous similar message [ 7098.447957] Lustre: server umount lustre-OST0000 complete [ 7102.944985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.947682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7102.961518] Lustre: Skipped 2 previous similar messages [ 7102.979632] Lustre: Skipped 1 previous similar message [ 7108.319842] Lustre: server umount lustre-MDT0000 complete [ 7109.094824] LustreError: 232594:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.108389] LustreError: 232594:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7112.307448] LustreError: 233327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861338 with bad export cookie 2983328842089138239 [ 7112.313358] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7112.323306] LustreError: 233327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7112.701564] Lustre: server umount lustre-MDT0001 complete [ 7124.293526] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7126.997941] Key type lgssc unregistered [ 7127.323439] LNet: 236291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7127.331190] LNetError: 236291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7127.357361] LNet: Removed LNI 192.168.203.115@tcp [ 7128.400702] Key type .llcrypt unregistered [ 7128.402340] Key type ._llcrypt unregistered [ 7150.442726] Key type ._llcrypt registered [ 7150.446195] Key type .llcrypt registered [ 7150.624858] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7152.121642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7152.180704] alg: No test for adler32 (adler32-zlib) [ 7153.474586] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7153.713435] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7155.383210] Key type lgssc registered [ 7156.405661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7426.808662] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7429.488804] Key type lgssc unregistered [ 7429.825491] LNet: 238328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7429.829911] LNetError: 238328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7429.871473] LNet: Removed LNI 192.168.203.115@tcp [ 7430.734514] Key type .llcrypt unregistered [ 7430.737669] Key type ._llcrypt unregistered [ 7460.157264] Key type ._llcrypt registered [ 7460.159553] Key type .llcrypt registered [ 7460.300655] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7479.044222] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7480.153311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7480.207503] alg: No test for adler32 (adler32-zlib) [ 7481.398754] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7481.688293] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7483.359165] Key type lgssc registered [ 7484.527494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7519.749581] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7249 sec ========== 16:15:43 (1787861743) [ 7521.771744] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:15:46 (1787861746) === [ 7525.805615] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:15:50 (1787861750) === [ 7569.380262] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7572.134791] Key type lgssc unregistered [ 7572.462480] LNet: 243568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7572.474761] LNetError: 243568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7572.488687] LNet: Removed LNI 192.168.203.115@tcp [ 7573.190843] Key type .llcrypt unregistered [ 7573.197783] Key type ._llcrypt unregistered