[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492096278 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003324] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007963] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.011064] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013059] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.023674] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028653] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031064] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032140] *** VALIDATE tmpfs *** [ 0.033490] *** VALIDATE proc *** [ 0.034291] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036323] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040023] Speculative Store Bypass: Vulnerable [ 0.043438] debug: unmapping init [mem 0xffffffffb3e59000-0xffffffffb3e60fff] [ 0.046219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047778] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048033] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054016] ... event mask: 000000070000000f [ 0.055330] rcu: Hierarchical SRCU implementation. [ 0.057624] smp: Bringing up secondary CPUs ... [ 0.058678] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.069181] smp: Brought up 1 node, 4 CPUs [ 0.071020] smpboot: Max logical packages: 1 [ 0.072016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.100389] node 0 deferred pages initialised in 26ms [ 0.104122] devtmpfs: initialized [ 0.106370] x86/mm: Memory block size: 128MB [ 0.109821] gcov: version magic: 0x41383552 [ 0.113329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.121533] pinctrl core: initialized pinctrl subsystem [ 0.123187] [ 0.124010] ************************************************************* [ 0.127012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128009] ** ** [ 0.131013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133012] ** ** [ 0.135045] ** This means that this kernel is built to expose internal ** [ 0.137008] ** IOMMU data structures, which may compromise security on ** [ 0.138008] ** your system. ** [ 0.140016] ** ** [ 0.142009] ** If you see this message and you are not debugging the ** [ 0.143007] ** kernel, report this immediately to your vendor! ** [ 0.145009] ** ** [ 0.146011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148015] ************************************************************* [ 0.149576] NET: Registered protocol family 16 [ 0.151316] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157091] cpuidle: using governor menu [ 0.158578] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160337] PCI: Using configuration type 1 for base access [ 0.161101] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.171119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175136] cryptd: max_cpu_qlen set to 1000 [ 0.177381] ACPI: Added _OSI(Module Device) [ 0.179016] ACPI: Added _OSI(Processor Device) [ 0.180019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.182013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194328] ACPI: Interpreter enabled [ 0.195063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197017] ACPI: Using IOAPIC for interrupt routing [ 0.198128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202500] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213090] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.218023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.221089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.225033] acpiphp: Slot [2] registered [ 0.226051] acpiphp: Slot [5] registered [ 0.227059] acpiphp: Slot [6] registered [ 0.228089] acpiphp: Slot [7] registered [ 0.229088] acpiphp: Slot [8] registered [ 0.231156] acpiphp: Slot [9] registered [ 0.232110] acpiphp: Slot [10] registered [ 0.234125] acpiphp: Slot [3] registered [ 0.235102] acpiphp: Slot [4] registered [ 0.236098] acpiphp: Slot [11] registered [ 0.238104] acpiphp: Slot [12] registered [ 0.239126] acpiphp: Slot [13] registered [ 0.241102] acpiphp: Slot [14] registered [ 0.242088] acpiphp: Slot [15] registered [ 0.243011] acpiphp: Slot [16] registered [ 0.243892] acpiphp: Slot [17] registered [ 0.244058] acpiphp: Slot [18] registered [ 0.244913] acpiphp: Slot [19] registered [ 0.246067] acpiphp: Slot [20] registered [ 0.246985] acpiphp: Slot [21] registered [ 0.248089] acpiphp: Slot [22] registered [ 0.249078] acpiphp: Slot [23] registered [ 0.250122] acpiphp: Slot [24] registered [ 0.252119] acpiphp: Slot [25] registered [ 0.253097] acpiphp: Slot [26] registered [ 0.254090] acpiphp: Slot [27] registered [ 0.256084] acpiphp: Slot [28] registered [ 0.257070] acpiphp: Slot [29] registered [ 0.258158] acpiphp: Slot [30] registered [ 0.260090] acpiphp: Slot [31] registered [ 0.261121] PCI host bridge to bus 0000:00 [ 0.262039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.267023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.269028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.275031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280072] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.283240] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.298547] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.303019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.305022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308581] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.311434] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.314054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.316899] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.323024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.335023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.342039] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351263] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.380030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.389915] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.397025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.406028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.426034] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439260] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.447013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.467025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.475984] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.484026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.495028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.515028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.526196] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.531017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.540024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.551019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.560907] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.566019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.571016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.582018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.592159] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.595682] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.598436] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.601518] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.603321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.609035] iommu: Default domain type: Passthrough [ 0.610499] SCSI subsystem initialized [ 0.612151] ACPI: bus type USB registered [ 0.614121] usbcore: registered new interface driver usbfs [ 0.616097] usbcore: registered new interface driver hub [ 0.618085] usbcore: registered new device driver usb [ 0.620206] pps_core: LinuxPPS API ver. 1 registered [ 0.622013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.625062] PTP clock support registered [ 0.627150] EDAC MC: Ver: 3.0.0 [ 0.629223] PCI: Using ACPI for IRQ routing [ 0.632033] NetLabel: Initializing [ 0.633012] NetLabel: domain hash size = 128 [ 0.635016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.637118] NetLabel: unlabeled traffic allowed by default [ 0.639123] vgaarb: loaded [ 0.640326] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.642031] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.651217] clocksource: Switched to clocksource kvm-clock [ 0.770389] VFS: Disk quotas dquot_6.6.0 [ 0.772020] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.774653] *** VALIDATE ramfs *** [ 0.775955] *** VALIDATE hugetlbfs *** [ 0.777539] pnp: PnP ACPI init [ 0.780348] pnp: PnP ACPI: found 6 devices [ 0.800863] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.803918] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.806100] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.808159] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.810660] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.813142] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.816093] NET: Registered protocol family 2 [ 0.818638] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.823418] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.827096] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.833226] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.837218] TCP: Hash tables configured (established 65536 bind 65536) [ 0.840465] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.843807] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.846748] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850448] NET: Registered protocol family 1 [ 0.853625] RPC: Registered named UNIX socket transport module. [ 0.856019] RPC: Registered udp transport module. [ 0.857820] RPC: Registered tcp transport module. [ 0.859435] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861772] NET: Registered protocol family 44 [ 0.863507] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.866260] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.868842] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871299] PCI: CLS 0 bytes, default 64 [ 0.873203] Unpacking initramfs... [ 2.241182] debug: unmapping init [mem 0xffff9e6f7cc54000-0xffff9e6f7ffbffff] [ 2.245341] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.247996] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.251653] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.705985] Initialise system trusted keyrings [ 2.707798] Key type blacklist registered [ 2.709769] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.719365] zbud: loaded [ 2.722753] *** VALIDATE nfs *** [ 2.724069] *** VALIDATE nfs4 *** [ 2.725649] pstore: using deflate compression [ 2.729319] Platform Keyring initialized [ 2.832312] NET: Registered protocol family 38 [ 2.833472] Key type asymmetric registered [ 2.834621] Asymmetric key parser 'x509' registered [ 2.836011] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.838040] io scheduler mq-deadline registered [ 2.839685] io scheduler kyber registered [ 2.841295] io scheduler bfq registered [ 2.843133] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.845905] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.848727] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.851505] ACPI: Power Button [PWRF] [ 2.856114] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.860899] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.869179] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.874439] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.887871] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.915453] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.944455] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.949384] Non-volatile memory driver v1.3 [ 2.951101] Linux agpgart interface v0.103 [ 2.980879] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.983691] vda: detected capacity change from 0 to 75071488 [ 2.998963] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.002073] vdb: detected capacity change from 0 to 1073741824 [ 3.019280] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.022729] vdc: detected capacity change from 0 to 2621440000 [ 3.037499] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.041251] vdd: detected capacity change from 0 to 2621440000 [ 3.058981] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.062236] vde: detected capacity change from 0 to 4294967296 [ 3.078577] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.081296] vdf: detected capacity change from 0 to 4294967296 [ 3.087892] libphy: Fixed MDIO Bus: probed [ 3.095097] usbcore: registered new interface driver usbserial_generic [ 3.097192] usbserial: USB Serial support registered for generic [ 3.099841] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.104297] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.106126] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.108747] mousedev: PS/2 mouse device common for all mice [ 3.112740] rtc_cmos 00:05: RTC can wake from S4 [ 3.115990] rtc_cmos 00:05: registered as rtc0 [ 3.117569] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.119130] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.126398] intel_pstate: CPU model not supported [ 3.127167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.132815] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.132853] hid: raw HID events driver (C) Jiri Kosina [ 3.138073] usbcore: registered new interface driver usbhid [ 3.140123] usbhid: USB HID core driver [ 3.141588] drop_monitor: Initializing network drop monitor service [ 3.143827] Initializing XFRM netlink socket [ 3.145759] NET: Registered protocol family 10 [ 3.148533] Segment Routing with IPv6 [ 3.149952] NET: Registered protocol family 17 [ 3.151740] mpls_gso: MPLS GSO support [ 3.156834] RAS: Correctable Errors collector initialized. [ 3.158989] AVX version of gcm_enc/dec engaged. [ 3.160938] AES CTR mode by8 optimization enabled [ 3.222711] sched_clock: Marking stable (3222681540, 0)->(4174216868, -951535328) [ 3.226077] registered taskstats version 1 [ 3.228059] Loading compiled-in X.509 certificates [ 3.230052] zswap: loaded using pool lzo/zbud [ 3.251786] Key type big_key registered [ 3.262890] Key type encrypted registered [ 3.264908] ima: No TPM chip found, activating TPM-bypass! [ 3.267041] ima: Allocated hash algorithm: sha1 [ 3.268559] ima: No architecture policies found [ 3.269903] evm: Initialising EVM extended attributes: [ 3.271149] evm: security.selinux [ 3.271788] evm: security.ima [ 3.272426] evm: security.capability [ 3.273269] evm: HMAC attrs: 0x1 [ 3.274963] rtc_cmos 00:05: setting system clock to 2026-08-27 11:39:10 UTC (1787830750) [ 3.280021] debug: unmapping init [mem 0xffffffffb4e03000-0xffffffffb4ffffff] [ 3.282171] debug: unmapping init [mem 0xffffffffb3b82000-0xffffffffb3e58fff] [ 3.290089] Write protecting the kernel read-only data: 28672k [ 3.292175] debug: unmapping init [mem 0xffffffffb2203000-0xffffffffb23fffff] [ 3.293792] debug: unmapping init [mem 0xffffffffb2b14000-0xffffffffb2bfffff] [ 3.323459] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.330577] systemd[1]: Detected virtualization kvm. [ 3.332210] systemd[1]: Detected architecture x86-64. [ 3.333778] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.360252] systemd[1]: No hostname configured. [ 3.361487] systemd[1]: Set hostname to . [ 3.362737] random: systemd: uninitialized urandom read (16 bytes read) [ 3.364627] systemd[1]: Initializing machine ID from random generator. [ 3.484866] random: systemd: uninitialized urandom read (16 bytes read) [ 3.487468] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.493151] random: systemd: uninitialized urandom read (16 bytes read) [ 3.495732] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.500765] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.062328] device-mapper: uevent: version 1.0.3 [ 4.064683] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.712393] random: fast init done [ 4.726696] virtio_net virtio0 ens2: renamed from eth0 [ 4.782370] scsi host0: ata_piix [ 4.828680] scsi host1: ata_piix [ 4.830095] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.832615] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.588365] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.777477] random: crng init done [ 9.779326] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.235831] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.493060] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.726195] SELinux: Disabled at runtime. [ 11.783893] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.793261] systemd[1]: Detected virtualization kvm. [ 11.795220] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.265397] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.268878] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.272855] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.276678] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.279327] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.288631] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.294346] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ 12.412104] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.711122] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.116405] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.187146] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.245566] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.270317] EDAC sbridge: Ver: 1.1.2 [ 15.004284] Key type dns_resolver registered [ 15.310798] NFS: Registering the id_resolver key type [ 15.314317] Key type id_resolver registered [ 15.315921] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg107-server login: [ 78.663926] libcfs: loading out-of-tree module taints kernel. [ 78.725227] Key type ._llcrypt registered [ 78.728019] Key type .llcrypt registered [ 78.866467] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 98.565093] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 100.455349] hrtimer: interrupt took 7838469 ns [ 100.575288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.599201] alg: No test for adler32 (adler32-zlib) [ 102.489721] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 103.262620] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 104.943208] Key type lgssc registered [ 106.138393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.342033] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.643584] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 184.607769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.654179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.907458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.944570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.000859] Lustre: lustre-MDT0000: new disk, initializing [ 186.061808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.073493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.281457] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.654246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.821864] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.898301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.909886] Lustre: Skipped 1 previous similar message [ 205.102424] Lustre: lustre-MDT0001: new disk, initializing [ 205.212183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.271179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.287988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.553326] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.947177] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.339680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.634844] Lustre: lustre-OST0000: new disk, initializing [ 226.637678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.642215] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.695848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.816600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.827755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.925270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.102746] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.879164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.035250] Lustre: lustre-OST0001: new disk, initializing [ 249.039532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.047738] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.114861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.395406] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.095961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.104566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.178859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.364317] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.308865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 289.565065] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing check_logdir /tmp/testlogs/ [ 294.972684] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing yml_node [ 300.054982] Lustre: DEBUG MARKER: Client: 2.17.57.60 [ 303.014423] Lustre: DEBUG MARKER: MDS: 2.17.57.60 [ 306.405451] Lustre: DEBUG MARKER: OSS: 2.17.57.60 [ 308.408972] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:44:13 EDT 2026 [ 327.817626] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 330.027142] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.091512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.104373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.107083] Lustre: Skipped 2 previous similar messages [ 351.200225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.212670] Lustre: Skipped 3 previous similar messages [ 352.502945] Lustre: server umount lustre-MDT0000 complete [ 356.625045] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831103 with bad export cookie 979522604593675320 [ 356.628721] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.635771] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 361.440117] LustreError: 8457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.440665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.459384] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.482818] LustreError: 8457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 361.542750] Lustre: Skipped 4 previous similar messages [ 363.207957] Lustre: server umount lustre-MDT0001 complete [ 368.471835] Lustre: server umount lustre-OST0000 complete [ 372.652594] Lustre: server umount lustre-OST0001 complete [ 380.997781] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 393.300032] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 432.655473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.882237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 432.899962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 432.945526] Lustre: lustre-MDT0000: new disk, initializing [ 433.015139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.035359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.996760] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 449.017485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.216813] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.273297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.275612] Lustre: Skipped 1 previous similar message [ 449.333160] Lustre: lustre-MDT0001: new disk, initializing [ 449.413433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.434870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 449.441320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.126498] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 464.425865] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 472.965786] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 481.507448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 481.835201] Lustre: lustre-OST0000: new disk, initializing [ 481.840090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 481.845954] Lustre: 17045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 481.932159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.810085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 483.825637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 483.960707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 490.735167] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 502.851257] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 507.791897] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 507.993548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 512.548398] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 512.710269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 519.663421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.668829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.671053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.678025] Lustre: Skipped 1 previous similar message [ 519.689212] Lustre: Skipped 2 previous similar messages [ 524.775043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.789124] Lustre: Skipped 1 previous similar message [ 528.863157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 529.068020] Lustre: server umount lustre-OST0000 complete [ 535.010528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.038688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 535.050256] Lustre: Skipped 1 previous similar message [ 547.296579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.525924] Lustre: server umount lustre-MDT0000 complete [ 550.369409] LustreError: 15091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.383599] LustreError: 15091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 550.924100] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831298 with bad export cookie 979522604593676860 [ 550.933127] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 550.937183] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.315221] Lustre: server umount lustre-MDT0001 complete [ 555.663878] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:48:20 (1787831300) [ 563.597773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.220740] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.321357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.777479] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 569.314773] LustreError: 19314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.435592] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.644512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.010739] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 594.429529] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 600.566227] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.990310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 614.266490] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 622.324306] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 633.832724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 633.840465] LustreError: Skipped 1 previous similar message [ 633.842831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.848698] Lustre: Skipped 1 previous similar message [ 633.853898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 633.861536] Lustre: Skipped 4 previous similar messages [ 638.550440] Lustre: server umount lustre-OST0000 complete [ 644.579773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.625543] Lustre: Skipped 2 previous similar messages [ 657.375150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 658.913316] LustreError: 20299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.944521] LustreError: 20299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 659.689880] Lustre: server umount lustre-MDT0000 complete [ 663.975246] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831411 with bad export cookie 979522604593677742 [ 663.985762] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 663.997926] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 664.448851] Lustre: server umount lustre-MDT0001 complete [ 679.247964] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 682.562772] Key type lgssc unregistered [ 682.810630] LNet: 22851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.816183] LNetError: 22851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.841764] LNet: Removed LNI 192.168.201.107@tcp [ 683.647639] Key type .llcrypt unregistered [ 683.649573] Key type ._llcrypt unregistered [ 709.062238] Key type ._llcrypt registered [ 709.064905] Key type .llcrypt registered [ 709.190910] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 710.282985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.392873] alg: No test for adler32 (adler32-zlib) [ 711.435122] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 711.731352] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 713.423309] Key type lgssc registered [ 714.634524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 723.456317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 723.484389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.950503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.594915] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 735.206455] LustreError: 24268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 735.229502] LustreError: 24268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 738.671186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.112857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.503339] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 752.912330] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 759.357673] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 768.401367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.802318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.849635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 776.078604] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 784.789811] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 804.832783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 804.838796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.846517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 804.849607] Lustre: Skipped 1 previous similar message [ 804.861215] Lustre: Skipped 1 previous similar message [ 807.554990] Lustre: server umount lustre-OST0000 complete [ 815.076413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.079625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 815.090604] Lustre: Skipped 1 previous similar message [ 815.104879] Lustre: Skipped 1 previous similar message [ 820.194573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.207159] Lustre: Skipped 1 previous similar message [ 825.313469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 825.324848] Lustre: Skipped 1 previous similar message [ 826.335489] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 826.690087] Lustre: server umount lustre-MDT0000 complete [ 830.432218] LustreError: 26722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.461732] LustreError: 26722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 832.132677] LustreError: 26111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831579 with bad export cookie 4140350204929254474 [ 832.164208] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.899740] Lustre: server umount lustre-MDT0001 complete [ 847.754470] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 850.592322] Key type lgssc unregistered [ 851.004801] LNet: 28250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.009600] LNetError: 28250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.028427] LNet: Removed LNI 192.168.201.107@tcp [ 852.108861] Key type .llcrypt unregistered [ 852.112413] Key type ._llcrypt unregistered [ 877.895777] Key type ._llcrypt registered [ 877.902765] Key type .llcrypt registered [ 878.055734] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 879.393374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.435617] alg: No test for adler32 (adler32-zlib) [ 880.514740] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 880.729595] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 882.415227] Key type lgssc registered [ 883.337730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.484503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.499519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.041834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.263526] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 902.627264] LustreError: 29665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.648808] LustreError: 29665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 906.466318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.913773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 911.055890] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 919.326892] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 925.739947] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 939.583336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 939.907219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.130252] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 954.027156] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 962.491161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 972.785718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.791831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 972.795938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.818151] Lustre: Skipped 1 previous similar message [ 972.820873] LustreError: Skipped 1 previous similar message [ 977.821303] Lustre: server umount lustre-OST0000 complete [ 983.009928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.030891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.033605] Lustre: Skipped 1 previous similar message [ 988.128610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.137658] Lustre: Skipped 2 previous similar messages [ 993.248928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.256557] Lustre: Skipped 1 previous similar message [ 995.296673] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 995.463688] Lustre: server umount lustre-MDT0000 complete [ 998.367869] LustreError: 29665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.387884] LustreError: 29665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 999.337054] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831746 with bad export cookie 1988691363320372693 [ 999.348053] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 999.349632] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.896808] Lustre: server umount lustre-MDT0001 complete [ 1013.713687] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1016.900753] Key type lgssc unregistered [ 1017.307281] LNet: 33569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.312822] LNetError: 33569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.374795] LNet: Removed LNI 192.168.201.107@tcp [ 1018.618992] Key type .llcrypt unregistered [ 1018.631287] Key type ._llcrypt unregistered [ 1046.683856] Key type ._llcrypt registered [ 1046.687305] Key type .llcrypt registered [ 1046.791835] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1047.985942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.063106] alg: No test for adler32 (adler32-zlib) [ 1049.239505] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1049.542313] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1051.247169] Key type lgssc registered [ 1052.681436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.591122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.632684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.359222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.848912] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1073.648464] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.677839] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1078.763649] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.762187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.145967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.345047] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1093.165430] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.786234] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1106.847277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.303551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.356453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1113.428067] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1122.008460] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.106294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.109559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.113681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.117520] LustreError: Skipped 1 previous similar message [ 1136.131026] Lustre: Skipped 1 previous similar message [ 1141.219712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.234472] Lustre: Skipped 1 previous similar message [ 1141.578438] Lustre: server umount lustre-OST0000 complete [ 1146.350063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.359157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.375548] Lustre: Skipped 2 previous similar messages [ 1151.457084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.466473] Lustre: Skipped 2 previous similar messages [ 1151.752439] Lustre: server umount lustre-MDT0000 complete [ 1155.438055] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831902 with bad export cookie 4021737604663781128 [ 1155.441623] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1155.453435] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1155.887261] Lustre: server umount lustre-MDT0001 complete [ 1168.951970] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1172.466562] Key type lgssc unregistered [ 1172.803705] LNet: 38561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.813399] LNetError: 38561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.832174] LNet: Removed LNI 192.168.201.107@tcp [ 1173.859139] Key type .llcrypt unregistered [ 1173.861802] Key type ._llcrypt unregistered [ 1199.402629] Key type ._llcrypt registered [ 1199.405633] Key type .llcrypt registered [ 1199.509622] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1200.660302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.823984] alg: No test for adler32 (adler32-zlib) [ 1201.900992] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1202.128409] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1203.783197] Key type lgssc registered [ 1204.868327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1213.385395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1213.433965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.074449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.423697] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1225.193464] LustreError: 39977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.215853] LustreError: 39977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1229.789675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.143396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1234.846340] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1243.488863] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1250.040706] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1259.108107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1259.570672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.619035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1268.963139] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1276.391729] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1283.382138] Lustre: lustre-OST0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1285.111855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.114675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.128391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.144280] Lustre: Skipped 1 previous similar message [ 1285.157403] Lustre: Skipped 1 previous similar message [ 1287.963523] Lustre: server umount lustre-OST0000 complete [ 1310.701455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.708232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.713725] Lustre: Skipped 1 previous similar message [ 1310.730362] Lustre: Skipped 1 previous similar message [ 1314.994330] Lustre: server umount lustre-MDT0000 complete [ 1315.817556] LustreError: 39977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.834194] LustreError: 39977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1318.852561] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832066 with bad export cookie 2798373827376945294 [ 1318.859052] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.862229] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1333.215148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1333.521649] Lustre: server umount lustre-MDT0001 complete [ 1346.967620] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1349.505788] Key type lgssc unregistered [ 1349.715928] LNet: 43702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.723694] LNetError: 43702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.746360] LNet: Removed LNI 192.168.201.107@tcp [ 1350.586124] Key type .llcrypt unregistered [ 1350.588261] Key type ._llcrypt unregistered [ 1376.696256] Key type ._llcrypt registered [ 1376.698564] Key type .llcrypt registered [ 1376.794394] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1377.715230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.742590] alg: No test for adler32 (adler32-zlib) [ 1379.014182] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1379.258105] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1380.944079] Key type lgssc registered [ 1382.708480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.460793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1392.494989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.305973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.184914] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1403.873833] LustreError: 45117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.882590] LustreError: 45117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1408.995982] LustreError: 45116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.612443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.267643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.514827] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1424.655860] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.524443] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1440.902267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.172467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.638534] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1456.645468] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.773342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1473.832739] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 1475.045570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.084892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.930774] Lustre: server umount lustre-MDT0000 complete [ 1478.924101] LustreError: 45112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.590351] LustreError: 45852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832226 with bad export cookie 10110733924404702211 [ 1479.598351] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1479.598577] LustreError: 45852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1480.166929] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.180399] Lustre: Skipped 2 previous similar messages [ 1480.194336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1480.203498] Lustre: Skipped 2 previous similar messages [ 1484.043744] LustreError: 45111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.071498] LustreError: 45111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1484.072391] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.7@tcp (stopping) [ 1486.179921] Lustre: server umount lustre-MDT0001 complete [ 1509.485441] Lustre: server umount lustre-OST0000 complete [ 1529.052542] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1532.279951] Key type lgssc unregistered [ 1532.535715] LNet: 49003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.540282] LNetError: 49003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.550902] LNet: Removed LNI 192.168.201.107@tcp [ 1533.162516] Key type .llcrypt unregistered [ 1533.165310] Key type ._llcrypt unregistered [ 1557.742913] Key type ._llcrypt registered [ 1557.745308] Key type .llcrypt registered [ 1557.859276] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1559.199558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.231971] alg: No test for adler32 (adler32-zlib) [ 1560.342411] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1560.629754] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1562.296415] Key type lgssc registered [ 1563.398497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1571.144218] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1571.184523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1586.912203] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1592.034404] LustreError: 50389:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 1618.719242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.646988] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1635.969873] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1803.953481] Lustre: server umount lustre-OST0000 complete [ 1832.459161] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1835.714046] Key type lgssc unregistered [ 1836.130588] LNet: 52091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.152782] LNetError: 52091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.222450] LNet: Removed LNI 192.168.201.107@tcp [ 1838.212088] Key type .llcrypt unregistered [ 1838.213334] Key type ._llcrypt unregistered [ 1866.324262] Key type ._llcrypt registered [ 1866.326786] Key type .llcrypt registered [ 1866.483902] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1868.146948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.170952] alg: No test for adler32 (adler32-zlib) [ 1869.407521] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1869.753567] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1871.479746] Key type lgssc registered [ 1872.776314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.893584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1884.908945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.577878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.272988] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1895.968968] LustreError: 53508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.996516] LustreError: 53508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1901.030125] LustreError: 53509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.802506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.359147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1907.596446] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1915.331991] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1922.500962] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1930.828564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.459291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.466265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1938.759472] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1949.920794] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1957.345927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.349453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.356036] Lustre: Skipped 1 previous similar message [ 1957.365968] Lustre: Skipped 1 previous similar message [ 1959.634455] Lustre: server umount lustre-OST0000 complete [ 1967.587705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.613620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.629421] Lustre: Skipped 1 previous similar message [ 1971.377769] Lustre: server umount lustre-MDT0000 complete [ 1972.706536] LustreError: 53508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.738609] LustreError: 53508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1976.047129] LustreError: 53489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832723 with bad export cookie 8959763896078500062 [ 1976.055647] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.068130] LustreError: 53489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1976.693737] Lustre: server umount lustre-MDT0001 complete [ 1989.875369] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1993.707754] Key type lgssc unregistered [ 1994.164711] LNet: 57042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.178834] LNetError: 57042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.204459] LNet: Removed LNI 192.168.201.107@tcp [ 1995.227444] Key type .llcrypt unregistered [ 1995.229772] Key type ._llcrypt unregistered [ 2022.069282] Key type ._llcrypt registered [ 2022.071181] Key type .llcrypt registered [ 2022.149291] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2023.335821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.370503] alg: No test for adler32 (adler32-zlib) [ 2024.473129] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2024.632827] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2026.271884] Key type lgssc registered [ 2027.220848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.910772] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.932313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2050.463645] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2055.583396] LustreError: 58429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 2082.338619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.437125] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2097.234806] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2104.288790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.775612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.860377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2109.933284] LustreError: 58454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.949349] LustreError: 58454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2110.437347] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2115.049263] LustreError: 58454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.133800] LustreError: 59465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.227403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.667166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2127.317280] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2135.773515] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.804982] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2145.765868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2145.771376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.779637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.300326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.304082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.321290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.102858] Lustre: server umount lustre-OST0000 complete [ 2172.907631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.926134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.926727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.944526] Lustre: Skipped 2 previous similar messages [ 2177.420113] Lustre: server umount lustre-MDT0000 complete [ 2178.022993] LustreError: 60957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.040139] LustreError: 60957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2181.244081] LustreError: 58435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832928 with bad export cookie 10148643706148600647 [ 2181.259376] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2187.534400] Lustre: server umount lustre-MDT0001 complete [ 2199.302696] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2202.462347] Key type lgssc unregistered [ 2202.836456] LNet: 62180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.847849] LNetError: 62180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.880301] LNet: Removed LNI 192.168.201.107@tcp [ 2204.002055] Key type .llcrypt unregistered [ 2204.004931] Key type ._llcrypt unregistered [ 2227.768223] Key type ._llcrypt registered [ 2227.773453] Key type .llcrypt registered [ 2227.895524] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2228.965509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.337935] alg: No test for adler32 (adler32-zlib) [ 2230.506825] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2230.949852] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2232.783238] Key type lgssc registered [ 2234.664598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.982595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.993475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.477731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.999620] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2258.918739] LustreError: 63576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.949980] LustreError: 63576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2264.035176] LustreError: 63577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.465408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.070302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2270.814590] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2278.853661] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2285.395939] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.626782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.285926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.324111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2298.937988] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2307.838846] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2327.017297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.018302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.026265] Lustre: Skipped 1 previous similar message [ 2327.026587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2327.045945] Lustre: Skipped 1 previous similar message [ 2332.137338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2332.160834] Lustre: Skipped 1 previous similar message [ 2337.250128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.262522] Lustre: Skipped 1 previous similar message [ 2339.807436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2339.949612] Lustre: server umount lustre-OST0000 complete [ 2347.490555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.493393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2347.503373] Lustre: Skipped 1 previous similar message [ 2347.522523] Lustre: Skipped 1 previous similar message [ 2350.615375] Lustre: server umount lustre-MDT0000 complete [ 2352.608351] LustreError: 63577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.629316] LustreError: 63577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2355.782357] LustreError: 63557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833103 with bad export cookie 17304795283269370565 [ 2355.788518] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2355.797155] LustreError: 63557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2356.313358] Lustre: server umount lustre-MDT0001 complete [ 2370.159513] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2373.563654] Key type lgssc unregistered [ 2374.002200] LNet: 67134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.011211] LNetError: 67134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.034829] LNet: Removed LNI 192.168.201.107@tcp [ 2375.029066] Key type .llcrypt unregistered [ 2375.034565] Key type ._llcrypt unregistered [ 2407.721291] Key type ._llcrypt registered [ 2407.725806] Key type .llcrypt registered [ 2407.833969] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2409.453177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.719810] alg: No test for adler32 (adler32-zlib) [ 2410.993266] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2411.474865] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2413.319313] Key type lgssc registered [ 2415.235754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2427.791364] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 08:19:32 (1787833172) [ 2436.296925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2436.340864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.925507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.694937] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2448.353756] LustreError: 68841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.390780] LustreError: 68841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2452.024245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.544799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.442653] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2465.057153] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2471.525943] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2478.898149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.248392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.316744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2485.587775] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2494.672599] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2504.471239] Lustre: Failing over lustre-MDT0000 [ 2504.846170] Lustre: server umount lustre-MDT0000 complete [ 2507.567706] LustreError: 71337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.257977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.268318] Lustre: Skipped 2 previous similar messages [ 2512.655534] LustreError: 71291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.670476] LustreError: 71291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2513.698508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.851387] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.993720] Lustre: *** cfs_fail_loc=135, val=0*** [ 2514.080433] LustreError: 71935:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2514.092592] LustreError: 71935:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -2 [ 2514.106604] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2514.106604] [ 2514.126324] LustreError: MGC192.168.201.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2514.142292] LustreError: 71923:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2514.147366] LustreError: 71923:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2514.156573] LustreError: 71923:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2514.162244] LustreError: 71923:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2514.186266] Lustre: server umount lustre-MDT0000 complete [ 2514.189783] LustreError: 71923:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2517.777535] LustreError: 71291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.802899] LustreError: 71291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2520.742068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.899938] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.101428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.150337] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2522.900385] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2525.954129] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2526.197689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2526.255265] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2526.313822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2536.418601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.425576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.436608] Lustre: Skipped 1 previous similar message [ 2540.250924] Lustre: server umount lustre-OST0000 complete [ 2546.657088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.700402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.709299] Lustre: Skipped 2 previous similar messages [ 2550.368446] Lustre: server umount lustre-MDT0000 complete [ 2551.786642] LustreError: 68841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.800647] LustreError: 68841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2553.942161] LustreError: 70687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833301 with bad export cookie 4945400209809499749 [ 2553.950639] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.951568] LustreError: 70687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2554.270559] Lustre: server umount lustre-MDT0001 complete [ 2566.439128] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2569.036434] Key type lgssc unregistered [ 2569.344923] LNet: 73717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.348976] LNetError: 73717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2570.415712] LNet: Removed LNI 192.168.201.107@tcp [ 2571.142660] Key type .llcrypt unregistered [ 2571.145435] Key type ._llcrypt unregistered [ 2596.210524] Key type ._llcrypt registered [ 2596.212898] Key type .llcrypt registered [ 2596.384761] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2597.359746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2597.508815] alg: No test for adler32 (adler32-zlib) [ 2598.537453] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2598.856863] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2600.559215] Key type lgssc registered [ 2602.125331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2610.872788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2610.917357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.810717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.400858] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2621.937364] LustreError: 75130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.959618] LustreError: 75130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2627.043251] LustreError: 75131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.293342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.642689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.668387] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2640.184614] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.936339] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2653.007471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.165397] LustreError: 76950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.391372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.570168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2660.853655] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2669.808950] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.066523] Lustre: Failing over lustre-MDT0000 [ 2676.515903] LustreError: 77430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.636636] Lustre: server umount lustre-MDT0000 complete [ 2679.273562] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.289403] Lustre: Skipped 1 previous similar message [ 2681.619766] LustreError: 75126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.654149] LustreError: 75126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2687.269464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.412397] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2687.622588] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2687.792797] LustreError: 78219:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2687.800480] LustreError: 78219:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -9 [ 2687.810420] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2687.810420] [ 2687.820126] LustreError: MGC192.168.201.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2687.832883] LustreError: 78207:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2687.840197] LustreError: 78207:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2687.848756] LustreError: 78207:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2687.856445] LustreError: 78207:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2687.884522] Lustre: server umount lustre-MDT0000 complete [ 2687.890890] LustreError: 78207:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2691.860480] LustreError: 75125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.882610] LustreError: 75125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2695.245330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.518464] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.971210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.041788] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2696.985720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2699.989366] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2701.314363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2701.349753] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2701.421481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2711.543581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.547115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.556114] Lustre: Skipped 2 previous similar messages [ 2716.651091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.654181] Lustre: Skipped 1 previous similar message [ 2717.165873] Lustre: server umount lustre-OST0000 complete [ 2726.880453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.911125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.745781] Lustre: server umount lustre-MDT0000 complete [ 2732.001929] LustreError: 76169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.039870] LustreError: 76169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2732.369027] LustreError: 75111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833479 with bad export cookie 2273508172957115703 [ 2732.372306] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.382027] LustreError: 75111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2732.774617] Lustre: server umount lustre-MDT0001 complete [ 2747.109492] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2750.800600] Key type lgssc unregistered [ 2751.201779] LNet: 80005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.218408] LNetError: 80005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2752.293808] LNet: Removed LNI 192.168.201.107@tcp [ 2753.196396] Key type .llcrypt unregistered [ 2753.198209] Key type ._llcrypt unregistered [ 2766.753762] Key type ._llcrypt registered [ 2766.755442] Key type .llcrypt registered [ 2766.852974] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2767.144355] Key type .llcrypt unregistered [ 2767.146391] Key type ._llcrypt unregistered [ 2792.005336] Key type ._llcrypt registered [ 2792.007774] Key type .llcrypt registered [ 2792.142430] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2793.279095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2793.460757] alg: No test for adler32 (adler32-zlib) [ 2794.487426] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2794.725437] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2796.375904] Key type lgssc registered [ 2797.320913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2803.972912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2803.987570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.587221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.033833] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2814.948398] LustreError: 81915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.963219] LustreError: 81915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2819.334176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.773436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.727627] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2832.819840] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2839.061638] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2846.954625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.336899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.420377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2854.715822] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2862.839593] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2875.368684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2875.369851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.392958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2875.399224] Lustre: Skipped 1 previous similar message [ 2875.413153] Lustre: Skipped 1 previous similar message [ 2880.485866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2880.506300] Lustre: Skipped 1 previous similar message [ 2880.990908] Lustre: server umount lustre-OST0000 complete [ 2885.602927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.622181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.630178] Lustre: Skipped 1 previous similar message [ 2890.721810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2890.728125] Lustre: Skipped 1 previous similar message [ 2895.842820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.848021] Lustre: Skipped 1 previous similar message [ 2898.399140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2898.600523] Lustre: server umount lustre-MDT0000 complete [ 2900.973973] LustreError: 81915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.992650] LustreError: 81915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2902.167513] LustreError: 84410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833649 with bad export cookie 4490126883885594408 [ 2902.169524] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.180988] LustreError: 84410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2902.525602] Lustre: server umount lustre-MDT0001 complete [ 2913.714978] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 2916.585644] Key type lgssc unregistered [ 2916.873262] LNet: 85454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2916.882391] LNetError: 85454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2916.895606] LNet: Removed LNI 192.168.201.107@tcp [ 2917.602914] Key type .llcrypt unregistered [ 2917.605189] Key type ._llcrypt unregistered [ 2942.636826] Key type ._llcrypt registered [ 2942.640749] Key type .llcrypt registered [ 2942.824175] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2944.349728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.382918] alg: No test for adler32 (adler32-zlib) [ 2945.396340] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2945.625792] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 2947.295638] Key type lgssc registered [ 2948.566440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2957.636650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2957.647112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.391396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.150887] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2969.570979] LustreError: 86866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.600080] LustreError: 86866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2974.691340] LustreError: 86885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.638447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.299298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.606366] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2989.786783] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2995.774268] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3003.841288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.533805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.208988] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3021.290732] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3029.109558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3039.712048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3039.718791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.722371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3039.753053] Lustre: Skipped 1 previous similar message [ 3039.783730] Lustre: Skipped 1 previous similar message [ 3044.642844] Lustre: server umount lustre-OST0000 complete [ 3049.953451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.957817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3049.979522] Lustre: Skipped 1 previous similar message [ 3049.987240] Lustre: Skipped 1 previous similar message [ 3054.457518] Lustre: server umount lustre-MDT0000 complete [ 3055.073052] LustreError: 86867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.086727] LustreError: 86867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3057.943567] LustreError: 86846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833805 with bad export cookie 10866668877975583539 [ 3057.954207] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3057.965235] LustreError: 86846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3058.364182] Lustre: server umount lustre-MDT0001 complete [ 3070.080401] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3072.809745] Key type lgssc unregistered [ 3073.112076] LNet: 90428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3073.122317] LNetError: 90428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3073.139909] LNet: Removed LNI 192.168.201.107@tcp [ 3074.030096] Key type .llcrypt unregistered [ 3074.033136] Key type ._llcrypt unregistered [ 3095.862365] Key type ._llcrypt registered [ 3095.865933] Key type .llcrypt registered [ 3095.956832] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3097.107880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3097.139806] alg: No test for adler32 (adler32-zlib) [ 3098.281929] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3098.480155] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3100.111150] Key type lgssc registered [ 3101.149718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3108.139948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3108.178485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.616957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.345779] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3119.083650] LustreError: 91847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.106734] LustreError: 91847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3122.323607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.674995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.434952] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3134.964788] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3141.147888] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3147.976385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3148.429413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.015483] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3154.578115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3161.437624] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3177.449074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.452981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3177.470801] Lustre: Skipped 1 previous similar message [ 3177.498119] Lustre: Skipped 1 previous similar message [ 3178.912935] Lustre: server umount lustre-OST0000 complete [ 3184.104105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.112806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.133681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.697076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.703417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.733172] Lustre: Skipped 1 previous similar message [ 3190.168560] Lustre: server umount lustre-MDT0000 complete [ 3192.801353] LustreError: 91843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.834063] LustreError: 91843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3194.513027] LustreError: 91827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833941 with bad export cookie 14358554204558583007 [ 3194.534585] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3194.550409] LustreError: 91827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3194.927258] Lustre: server umount lustre-MDT0001 complete [ 3206.216558] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3209.276780] Key type lgssc unregistered [ 3209.567315] LNet: 95403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3209.573951] LNetError: 95403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3210.599189] LNet: Removed LNI 192.168.201.107@tcp [ 3211.646618] Key type .llcrypt unregistered [ 3211.649120] Key type ._llcrypt unregistered [ 3239.358812] Key type ._llcrypt registered [ 3239.360859] Key type .llcrypt registered [ 3239.471754] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3240.471085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3240.498333] alg: No test for adler32 (adler32-zlib) [ 3241.658479] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3241.876889] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3243.527224] Key type lgssc registered [ 3244.793949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3252.613178] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3252.630229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.191522] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3273.311337] LustreError: 96789:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 3299.935200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.721936] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3314.226575] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3320.210533] Lustre: server umount lustre-OST0000 complete [ 3328.692984] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:34:34 (1787834074) [ 3336.519296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.931807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.537456] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3342.304516] LustreError: 98428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.329158] LustreError: 98428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3347.428411] LustreError: 98429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.474766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.766455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3354.546368] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3362.096656] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3367.169684] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3373.906752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3374.246182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.273391] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3382.459169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3388.032454] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3401.184338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.193760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.219876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.232398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3403.236028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.236946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.276631] Lustre: Skipped 1 previous similar message [ 3406.933845] Lustre: server umount lustre-OST0000 complete [ 3411.428992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3411.438508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.453615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.472778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.473759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.459533] Lustre: server umount lustre-MDT0000 complete [ 3418.592165] LustreError: 98423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.606382] LustreError: 98423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3420.178649] LustreError: 98408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834167 with bad export cookie 1824276260741906813 [ 3420.192775] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.628181] Lustre: server umount lustre-MDT0001 complete [ 3432.127155] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3435.201833] Key type lgssc unregistered [ 3435.514965] LNet: 102127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3435.520926] LNetError: 102127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3436.584739] LNet: Removed LNI 192.168.201.107@tcp [ 3437.505583] Key type .llcrypt unregistered [ 3437.511703] Key type ._llcrypt unregistered [ 3481.183291] Key type ._llcrypt registered [ 3481.188539] Key type .llcrypt registered [ 3481.286875] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3482.459585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.515563] alg: No test for adler32 (adler32-zlib) [ 3483.550824] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3483.726135] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3485.359544] Key type lgssc registered [ 3486.860883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3495.730225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3495.764593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.251587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.004532] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3507.686678] LustreError: 104314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.709614] LustreError: 104314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3512.805799] LustreError: 104315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.212860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.781852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.471433] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3529.163805] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3535.967516] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3543.867057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.327861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.223307] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3559.552395] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3566.927743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3580.384840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.391077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.411405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3582.433771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3582.444260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.468782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3582.479704] Lustre: Skipped 1 previous similar message [ 3585.907925] Lustre: server umount lustre-OST0000 complete [ 3591.648963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3591.661774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.681649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.720472] Lustre: server umount lustre-MDT0000 complete [ 3597.796075] LustreError: 105042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.840995] LustreError: 105042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3600.936709] LustreError: 104295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834348 with bad export cookie 2904318611745944983 [ 3600.938882] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.947148] LustreError: 104295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3601.251292] Lustre: server umount lustre-MDT0001 complete [ 3614.784913] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3617.662669] Key type lgssc unregistered [ 3617.918543] LNet: 107879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3617.921909] LNetError: 107879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3617.933588] LNet: Removed LNI 192.168.201.107@tcp [ 3618.640245] Key type .llcrypt unregistered [ 3618.642768] Key type ._llcrypt unregistered [ 3641.306904] Key type ._llcrypt registered [ 3641.310599] Key type .llcrypt registered [ 3641.423733] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3642.928470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3642.977615] alg: No test for adler32 (adler32-zlib) [ 3644.232927] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3644.489376] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3646.207344] Key type lgssc registered [ 3647.560694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3655.323986] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3655.342800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3670.880743] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3676.001762] LustreError: 109122:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 3702.687287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.345204] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3718.087477] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3724.357496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.448895] LustreError: 108459:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000061e66b73 x1874680234379520/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3724.603662] LustreError: MGC192.168.201.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3724.613992] LustreError: 110147:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3724.620224] LustreError: 110147:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3724.629101] LustreError: 110147:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3724.639837] LustreError: 109130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834471 with bad export cookie 14273207094629463904 [ 3724.655709] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.701619] Lustre: server umount lustre-MDT0000 complete [ 3724.703753] LustreError: 110147:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3733.025865] Lustre: server umount lustre-OST0000 complete [ 3743.273668] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3753.791451] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3790.277336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.529188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3790.612885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3790.735921] Lustre: lustre-MDT0000: new disk, initializing [ 3790.861971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.879863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3796.079287] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3808.244301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.409938] Lustre: 112943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3808.463693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3808.468215] Lustre: Skipped 1 previous similar message [ 3808.572162] Lustre: lustre-MDT0001: new disk, initializing [ 3808.668146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3808.719416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3808.745702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3814.482902] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3824.568947] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3830.498797] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3836.637381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3836.866363] Lustre: lustre-OST0000: new disk, initializing [ 3836.869806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3836.876876] Lustre: 114910:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3836.975544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.586926] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3845.724917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3845.737953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3845.893809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3854.569534] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3860.829138] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3861.141689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3866.113802] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3866.322727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3871.719090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3871.721447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.726622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.728503] LustreError: Skipped 1 previous similar message [ 3871.768909] Lustre: Skipped 1 previous similar message [ 3874.415662] Lustre: server umount lustre-OST0000 complete [ 3880.429526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3880.435242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.448496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3880.452576] Lustre: Skipped 1 previous similar message [ 3881.953631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.954348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.652805] Lustre: server umount lustre-MDT0000 complete [ 3888.302604] LustreError: 112936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834635 with bad export cookie 14273207094629464002 [ 3888.303472] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3888.310066] LustreError: 112936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3888.722485] Lustre: server umount lustre-MDT0001 complete [ 3897.001111] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 08:44:02 (1787834642) [ 3900.247548] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3917.425108] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3927.722352] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3966.818065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.194546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3967.224358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3967.378374] Lustre: lustre-MDT0000: new disk, initializing [ 3967.467470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.483598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3972.853428] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3985.787037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.952170] Lustre: 119820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3985.992538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3985.996267] Lustre: Skipped 1 previous similar message [ 3986.081480] Lustre: lustre-MDT0001: new disk, initializing [ 3986.148462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.170811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3986.177069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3990.762958] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4000.917931] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4006.919537] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.482120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.781227] Lustre: lustre-OST0000: new disk, initializing [ 4013.784681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4013.789979] Lustre: 121788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4013.879555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.835983] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4020.867807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.883864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4021.022434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4029.660721] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4034.861448] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4035.150561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4040.317909] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4040.632793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4046.825129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.828511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.829854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.829859] Lustre: Skipped 1 previous similar message [ 4046.834099] Lustre: Skipped 1 previous similar message [ 4046.866451] LustreError: Skipped 1 previous similar message [ 4049.036845] Lustre: server umount lustre-OST0000 complete [ 4052.960392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4052.968791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.993575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.001617] Lustre: Skipped 1 previous similar message [ 4058.940774] Lustre: server umount lustre-MDT0000 complete [ 4062.175919] LustreError: 121044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.194106] LustreError: 121044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4062.809307] LustreError: 119814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834810 with bad export cookie 14273207094629464884 [ 4062.811232] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.828746] LustreError: 119814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4063.080387] Lustre: server umount lustre-MDT0001 complete [ 4071.060264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.535807] LustreError: 124004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.610460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.263536] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4077.032418] LustreError: 124005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.122429] LustreError: 124004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.649051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.277470] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4099.127811] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4106.676029] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4113.532731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.808748] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4127.738897] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4138.238472] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4147.177587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4147.183667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.191822] Lustre: Skipped 2 previous similar messages [ 4147.198065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.201813] Lustre: Skipped 3 previous similar messages [ 4148.760583] Lustre: server umount lustre-OST0000 complete [ 4156.897644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4156.905939] LustreError: Skipped 1 previous similar message [ 4159.229535] Lustre: server umount lustre-MDT0000 complete [ 4162.529518] LustreError: 124001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.545168] LustreError: 124001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4163.170596] LustreError: 123984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834910 with bad export cookie 14273207094629465766 [ 4163.179208] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4163.623614] Lustre: server umount lustre-MDT0001 complete [ 4175.818280] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4178.374726] Key type lgssc unregistered [ 4178.617786] LNet: 127638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4178.624247] LNetError: 127638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4178.636604] LNet: Removed LNI 192.168.201.107@tcp [ 4179.245676] Key type .llcrypt unregistered [ 4179.248051] Key type ._llcrypt unregistered [ 4198.906554] Key type ._llcrypt registered [ 4198.911175] Key type .llcrypt registered [ 4199.024374] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 4214.926846] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4216.044241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4216.101058] alg: No test for adler32 (adler32-zlib) [ 4217.207945] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 4217.451536] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4219.119187] Key type lgssc registered [ 4220.257428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4252.799712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4252.822124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.084982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4254.129376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4254.188950] Lustre: lustre-MDT0000: new disk, initializing [ 4254.240733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.252604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4257.682749] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4267.793340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.923765] Lustre: 130793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4267.986149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4267.990569] Lustre: Skipped 1 previous similar message [ 4268.060389] Lustre: lustre-MDT0001: new disk, initializing [ 4268.151770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4268.201067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4268.214634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4272.126994] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4281.499740] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4287.894623] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4294.601588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.893949] Lustre: lustre-OST0000: new disk, initializing [ 4294.903693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4294.922635] Lustre: 132759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4295.083034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.986106] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4304.964424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4304.970903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4305.066443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4310.850523] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4316.245680] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4316.605899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4320.868512] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4321.111398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4325.859074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4325.861773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.870770] LustreError: Skipped 1 previous similar message [ 4325.874443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4325.888196] Lustre: Skipped 1 previous similar message [ 4325.913565] Lustre: Skipped 1 previous similar message [ 4329.233975] Lustre: server umount lustre-OST0000 complete [ 4334.562236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4334.572278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.603239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4336.105112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4336.109381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.983929] Lustre: server umount lustre-MDT0000 complete [ 4344.364778] LustreError: 130782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835091 with bad export cookie 6949522312851172125 [ 4344.370368] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.370682] LustreError: 130782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4344.721991] Lustre: server umount lustre-MDT0001 complete [ 4354.251893] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:51:40 (1787835100) [ 4361.723183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.145362] LustreError: 135174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.163849] LustreError: 135174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4362.266332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.336105] LustreError: 135175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.392628] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4372.450773] LustreError: 135174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.036561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4375.322321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4379.453442] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4386.291260] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4391.396401] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4396.000310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4396.009527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.025730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4396.031211] Lustre: Skipped 1 previous similar message [ 4399.612105] Lustre: server umount lustre-MDT0000 complete [ 4400.099907] LustreError: 135170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.115133] LustreError: 135170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4403.267399] LustreError: 135156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835150 with bad export cookie 6949522312851173007 [ 4403.272625] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.278037] LustreError: 135156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4403.720895] Lustre: server umount lustre-MDT0001 complete [ 4412.504794] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:52:38 (1787835158) [ 4419.001409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.530891] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4439.647433] LustreError: 137588:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 4465.311641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.829793] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4478.754438] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4481.219410] Lustre: server umount lustre-OST0000 complete [ 4491.836163] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:53:57 (1787835237) [ 4498.986965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.473289] LustreError: 139036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.563582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.897551] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4509.667728] LustreError: 139037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.692074] LustreError: 139037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4514.382319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.798488] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4528.271685] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4535.146951] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4542.879269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4543.253973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4543.258187] Lustre: Skipped 1 previous similar message [ 4548.747388] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4556.124198] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4570.097127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.108991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4570.115345] Lustre: Skipped 3 previous similar messages [ 4570.139426] Lustre: Skipped 4 previous similar messages [ 4575.012936] Lustre: server umount lustre-MDT0000 complete [ 4575.210505] LustreError: 140949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.240412] LustreError: 140949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4578.889612] LustreError: 139018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835326 with bad export cookie 6949522312851173672 [ 4578.910300] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4578.920702] LustreError: 139018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4579.332606] Lustre: server umount lustre-MDT0001 complete [ 4592.123766] Lustre: server umount lustre-OST0000 complete [ 4599.803338] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:55:45 (1787835345) [ 4606.016478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.152108] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4611.562834] LustreError: 142710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.575895] LustreError: 142710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4618.774346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.149400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4619.159155] Lustre: Skipped 1 previous similar message [ 4623.301297] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4630.451340] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4636.217522] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4642.235992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.432152] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4650.815758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4655.807451] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4660.252795] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4660.461744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4663.868385] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4664.072366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4666.339972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4666.347447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4666.351521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4666.366899] Lustre: Skipped 1 previous similar message [ 4680.159519] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4680.292943] Lustre: server umount lustre-OST0000 complete [ 4685.795453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4685.805077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.829795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4685.831662] Lustre: Skipped 5 previous similar messages [ 4690.568325] Lustre: server umount lustre-MDT0000 complete [ 4691.939537] LustreError: 142706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.963266] LustreError: 142706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4694.201641] LustreError: 142690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835441 with bad export cookie 6949522312851174743 [ 4694.215914] LustreError: 142690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4694.235566] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4694.480371] Lustre: server umount lustre-MDT0001 complete [ 4702.793914] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:57:28 (1787835448) [ 4708.425605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4724.063770] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4729.183651] LustreError: 146865:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 4754.911637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.924089] Lustre: Skipped 1 previous similar message [ 4761.369130] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4770.083094] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4778.475893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.855377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4783.970951] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4792.261216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.431381] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4805.274187] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4811.869760] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4816.853704] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4817.058292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4821.736885] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4821.998685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4828.649299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4828.650952] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.655906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4828.664498] Lustre: Skipped 2 previous similar messages [ 4828.671477] Lustre: Skipped 2 previous similar messages [ 4838.367252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4838.525690] Lustre: server umount lustre-MDT0000 complete [ 4838.882903] LustreError: 147903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4838.907082] LustreError: 147903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4842.283268] LustreError: 146871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835589 with bad export cookie 6949522312851175765 [ 4842.295121] LustreError: 146871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.302164] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4842.617070] Lustre: server umount lustre-MDT0001 complete [ 4856.816712] Lustre: server umount lustre-OST0000 complete [ 4865.745437] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:00:11 (1787835611) [ 4871.810210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4887.391584] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4892.512298] LustreError: 151062:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 4924.241947] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4931.554654] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4938.255131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.707091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4943.135786] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4952.399927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.577616] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4969.596152] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4975.601915] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4982.356676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4982.534296] Lustre: lustre-OST0001: new disk, initializing [ 4982.536880] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4982.540950] Lustre: 153946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4988.473718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4988.481087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4988.591702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4989.441991] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4999.805592] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5005.602433] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5005.846529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5010.769438] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5011.120855] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5014.498293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.509607] Lustre: Skipped 1 previous similar message [ 5014.520907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.531307] Lustre: Skipped 6 previous similar messages [ 5019.286223] Lustre: server umount lustre-OST0000 complete [ 5024.758529] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5030.195333] Lustre: server umount lustre-OST0001 complete [ 5048.287141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5048.497162] Lustre: server umount lustre-MDT0000 complete [ 5051.931899] LustreError: 151068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835799 with bad export cookie 6949522312851176717 [ 5051.940363] LustreError: 151068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5051.941167] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.291308] Lustre: server umount lustre-MDT0001 complete [ 5082.148641] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:03:47 (1787835827) [ 5084.616192] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5086.699271] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5089.162398] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:03:54 (1787835834) [ 5096.961614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.304924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5097.360373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5097.610033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.616102] Lustre: Skipped 6 previous similar messages [ 5102.920633] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5112.265411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.385789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5112.417855] Lustre: 157508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5116.723453] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5126.413446] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5133.119489] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5142.344596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5142.733597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5151.528432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5151.740531] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5161.365257] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5165.969798] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5166.170698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5171.636557] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5171.997953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5177.332598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.342690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5177.365888] Lustre: Skipped 5 previous similar messages [ 5177.371116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5177.371126] Lustre: Skipped 10 previous similar messages [ 5177.428093] LustreError: Skipped 2 previous similar messages [ 5180.780853] Lustre: server umount lustre-OST0000 complete [ 5209.931578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5218.046891] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.070915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.072081] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5218.124518] Lustre: Skipped 1 previous similar message [ 5220.388789] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5230.811751] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5237.404280] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5237.669251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5245.002858] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5245.280922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5253.773938] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b8403539800.ost_server_uuid 50 [ 5256.911430] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b8403539800.ost_server_uuid in IDLE state after 0 sec [ 5257.478969] LustreError: 161331:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.7@tcp) returned error from blocking AST (req@00000000aba036a8 x1874680835485312 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000017c9d8f7/0x6071a9b930254667 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.7@tcp remote: 0x7bf73039ec9071d8 expref: 5 pid: 157515 timeout: 5357 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5257.544474] LustreError: lustre-MDT0000: A client on nid 192.168.201.7@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5257.567590] LustreError: 157504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.7@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000017c9d8f7/0x6071a9b930254667 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.7@tcp remote: 0x7bf73039ec9071d8 expref: 6 pid: 157515 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5262.830598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5268.747253] Lustre: server umount lustre-OST0000 complete [ 5280.742841] Lustre: server umount lustre-MDT0000 complete [ 5283.809040] LustreError: 157519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.846670] LustreError: 157519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5285.186024] LustreError: 157501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836032 with bad export cookie 6949522312851177900 [ 5285.192936] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5285.200501] LustreError: 157501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5291.959722] Lustre: server umount lustre-MDT0001 complete [ 5307.654061] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5312.241045] Key type lgssc unregistered [ 5312.654910] LNet: 163631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5312.663504] LNetError: 163631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5312.684468] LNet: Removed LNI 192.168.201.107@tcp [ 5314.095179] Key type .llcrypt unregistered [ 5314.096582] Key type ._llcrypt unregistered [ 5344.865912] Key type ._llcrypt registered [ 5344.871209] Key type .llcrypt registered [ 5345.080985] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5346.789125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5346.823043] alg: No test for adler32 (adler32-zlib) [ 5347.772914] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 5347.958725] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5349.631251] Key type lgssc registered [ 5351.390048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5359.315869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5359.345900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.788493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.352726] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5370.363307] LustreError: 165050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5370.382549] LustreError: 165050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5374.160224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5374.686561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5379.661091] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5387.785692] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5392.984216] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5401.785378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.259742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.397723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5410.947154] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5424.778882] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5433.004565] Lustre: Failing over lustre-MDT0000 [ 5433.160880] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 5433.738790] Lustre: server umount lustre-MDT0000 complete [ 5434.339399] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.359049] LustreError: 166054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.380987] LustreError: 166054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5439.459281] LustreError: 165051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.479450] LustreError: 165051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5444.578917] LustreError: 165047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.635667] LustreError: 165047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5446.699222] LustreError: 165032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.7@tcp arrived at 1787836193 with bad export cookie 9173108323798596490 [ 5446.730194] LustreError: 165032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5449.715058] LustreError: 165050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.747693] LustreError: 165050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5450.729132] Lustre: 164362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836181/real 1787836181] req@000000004233bec4 x1874682021180416/t0(0) o400->MGC192.168.201.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787836197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5450.772182] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.536529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.970828] LustreError: 165051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.008303] LustreError: 165051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5461.293679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5461.351663] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5466.600426] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5466.618388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5466.626835] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5468.900215] Lustre: lustre-MDT0000: Denying connection for new client 1d6715be-47bd-47e0-ad24-aba8dec51522 (at 192.168.201.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5480.421358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5480.433242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.444363] Lustre: Skipped 2 previous similar messages [ 5480.457308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5481.954810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.966679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5481.977456] Lustre: Skipped 1 previous similar message [ 5483.444250] Lustre: server umount lustre-OST0000 complete [ 5487.700912] LustreError: 168785:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5487.709373] Lustre: 168134:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5487.720320] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5487.745367] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5487.759824] LustreError: 168134:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000e535482 x1874682021201536/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5487.780228] LustreError: 168134:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5487.788878] LustreError: 168134:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5487.801680] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5488.124717] Lustre: server umount lustre-MDT0000 complete [ 5492.196816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.215202] LustreError: 165046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.465206] LustreError: 165784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836239 with bad export cookie 9173108323798596588 [ 5492.482822] LustreError: 165784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5492.483307] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5494.987723] Lustre: server umount lustre-MDT0001 complete [ 5507.119978] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5510.002409] Key type lgssc unregistered [ 5510.343787] LNet: 169540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5510.352806] LNetError: 169540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5511.401487] LNet: Removed LNI 192.168.201.107@tcp [ 5512.313115] Key type .llcrypt unregistered [ 5512.320156] Key type ._llcrypt unregistered [ 5539.840210] Key type ._llcrypt registered [ 5539.844690] Key type .llcrypt registered [ 5539.998412] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5541.460758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5541.526958] alg: No test for adler32 (adler32-zlib) [ 5542.624368] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 5543.096319] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5544.935195] Key type lgssc registered [ 5546.087319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5554.379432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5554.388915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.186428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.533344] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5566.433204] LustreError: 170955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.463843] LustreError: 170955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5571.604780] LustreError: 170956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.301810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.964734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5579.768414] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5587.693962] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5593.917919] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.776955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.472635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.969995] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5620.087711] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5625.975329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5631.457060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5631.468957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.472767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5631.481839] Lustre: Skipped 1 previous similar message [ 5631.509113] Lustre: Skipped 1 previous similar message [ 5636.581973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5637.712951] Lustre: server umount lustre-OST0000 complete [ 5646.817052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5646.822466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.841696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5646.841704] Lustre: Skipped 1 previous similar message [ 5646.893322] Lustre: Skipped 1 previous similar message [ 5648.963486] Lustre: server umount lustre-MDT0000 complete [ 5651.940180] LustreError: 170949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.954221] LustreError: 170949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5653.420526] LustreError: 170935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836400 with bad export cookie 11377117449083343497 [ 5653.429726] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.442582] LustreError: 170935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5653.804489] Lustre: server umount lustre-MDT0001 complete [ 5666.766295] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5669.970850] Key type lgssc unregistered [ 5670.242703] LNet: 174493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5670.248982] LNetError: 174493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5671.281388] LNet: Removed LNI 192.168.201.107@tcp [ 5672.294920] Key type .llcrypt unregistered [ 5672.296746] Key type ._llcrypt unregistered [ 5712.083542] Key type ._llcrypt registered [ 5712.086386] Key type .llcrypt registered [ 5712.191456] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5713.836418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5713.875547] alg: No test for adler32 (adler32-zlib) [ 5715.016693] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 5715.270484] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5716.895188] Key type lgssc registered [ 5718.147696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5727.126541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5727.146524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.644223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.247732] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5737.960504] LustreError: 176198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.994302] LustreError: 176198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5743.079749] LustreError: 176199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.046423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.471221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5750.726909] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5759.947818] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5767.526418] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5775.228033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5775.534484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.883610] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5791.879436] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5798.071607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5808.609600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.616990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.624152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.644475] Lustre: Skipped 1 previous similar message [ 5808.658887] Lustre: Skipped 1 previous similar message [ 5813.730461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.749184] Lustre: Skipped 1 previous similar message [ 5818.866613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.873264] Lustre: Skipped 1 previous similar message [ 5820.383177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5820.756894] Lustre: server umount lustre-OST0000 complete [ 5827.552383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5827.556539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.600107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.093395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.991028] Lustre: server umount lustre-MDT0000 complete [ 5834.209224] LustreError: 176199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.241411] LustreError: 176199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5835.891425] LustreError: 176178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836583 with bad export cookie 532106238473364695 [ 5835.900160] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.917663] LustreError: 176178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5836.626827] Lustre: server umount lustre-MDT0001 complete [ 5849.801920] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5853.074721] Key type lgssc unregistered [ 5853.343800] LNet: 179761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5853.354275] LNetError: 179761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5854.380214] LNet: Removed LNI 192.168.201.107@tcp [ 5855.509207] Key type .llcrypt unregistered [ 5855.510631] Key type ._llcrypt unregistered [ 5880.527427] Key type ._llcrypt registered [ 5880.529568] Key type .llcrypt registered [ 5880.715287] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5882.765842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.798622] alg: No test for adler32 (adler32-zlib) [ 5884.082818] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 5884.286041] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5885.984112] Key type lgssc registered [ 5887.209085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5899.754393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5899.784691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.257684] Lustre: *** cfs_fail_loc=135, val=0*** [ 5901.421723] LustreError: 181286:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5901.425494] LustreError: 181286:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -2 [ 5901.433995] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5901.433995] [ 5901.454446] LustreError: MGC192.168.201.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5901.472423] LustreError: 181247:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5901.481028] LustreError: 181247:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5901.491268] LustreError: 181247:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5901.495804] LustreError: 181247:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5901.549895] Lustre: server umount lustre-MDT0000 complete [ 5901.551887] LustreError: 181247:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5910.030306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.667899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.157745] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5920.229801] LustreError: 181659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.252109] LustreError: 181659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5925.346667] LustreError: 181658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.567578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.982705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5933.496917] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5942.737697] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5948.246643] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5952.482563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5952.485189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5952.498917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5953.572174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5953.583102] Lustre: Skipped 1 previous similar message [ 5956.891450] Lustre: server umount lustre-MDT0000 complete [ 5958.640468] LustreError: 181659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.659364] LustreError: 181659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5960.420476] LustreError: 181640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836707 with bad export cookie 17252959600030602554 [ 5960.430582] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5960.771528] Lustre: server umount lustre-MDT0001 complete [ 5971.986774] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5975.016813] Key type lgssc unregistered [ 5975.317357] LNet: 183984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5975.329738] LNetError: 183984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5975.342672] LNet: Removed LNI 192.168.201.107@tcp [ 5976.319262] Key type .llcrypt unregistered [ 5976.322688] Key type ._llcrypt unregistered [ 6003.653753] Key type ._llcrypt registered [ 6003.656729] Key type .llcrypt registered [ 6003.799749] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6004.118976] Key type .llcrypt unregistered [ 6004.121377] Key type ._llcrypt unregistered [ 6025.289788] Key type ._llcrypt registered [ 6025.292063] Key type .llcrypt registered [ 6025.414600] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6026.508223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6026.608667] alg: No test for adler32 (adler32-zlib) [ 6027.582126] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6027.759826] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6029.464166] Key type lgssc registered [ 6030.839807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6038.618963] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6038.632532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6054.175992] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6059.295812] LustreError: 186122:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 6085.983213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.175714] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6100.043521] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6106.422771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.844238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.894611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6110.885716] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6112.225500] LustreError: 186148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.239485] LustreError: 186148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6117.348474] LustreError: 186147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.771584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.085608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6123.231536] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6129.758569] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6136.308727] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6139.880913] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6148.064484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.068540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.077182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6149.600387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6149.620972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.642308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.137823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.143198] Lustre: Skipped 1 previous similar message [ 6156.265598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.280113] Lustre: Skipped 1 previous similar message [ 6161.417412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6161.433887] Lustre: Skipped 1 previous similar message [ 6162.399183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6162.617532] Lustre: server umount lustre-OST0000 complete [ 6170.081220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6170.089759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.106128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6172.917452] Lustre: server umount lustre-MDT0000 complete [ 6176.745499] LustreError: 187162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.768099] LustreError: 187162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6176.957530] LustreError: 186128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836924 with bad export cookie 13953119803756739318 [ 6176.964233] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6176.969685] LustreError: 186128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6177.301995] Lustre: server umount lustre-MDT0001 complete [ 6189.223809] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6191.496514] Key type lgssc unregistered [ 6191.703758] LNet: 190043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6191.709070] LNetError: 190043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6192.744560] LNet: Removed LNI 192.168.201.107@tcp [ 6193.576888] Key type .llcrypt unregistered [ 6193.579554] Key type ._llcrypt unregistered [ 6225.396275] Key type ._llcrypt registered [ 6225.400505] Key type .llcrypt registered [ 6225.523767] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6226.556587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6226.632501] alg: No test for adler32 (adler32-zlib) [ 6227.857905] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6228.054920] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6229.711942] Key type lgssc registered [ 6230.587766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6239.638288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6239.667211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.214706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.612610] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6251.489340] LustreError: 191457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6251.524816] LustreError: 191457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6254.574878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.053180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.504735] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6267.306378] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6272.857717] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.490635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6279.771651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.490917] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6285.941741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6292.718848] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.180695] Lustre: Failing over lustre-MDT0000 [ 6297.479454] Lustre: server umount lustre-MDT0000 complete [ 6299.455783] LustreError: 192466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.152726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6301.167606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.188802] LustreError: 192467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.530330] LustreError: 191452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.547196] LustreError: 191452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6309.647282] LustreError: 191452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.684229] LustreError: 191452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6316.931108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.967243] Lustre: 190768:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837048/real 1787837048] req@00000000a39f9b50 x1874682943923456/t0(0) o400->MGC192.168.201.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787837064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6316.994663] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6319.882384] LustreError: 191453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.903970] LustreError: 191453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6327.285387] LustreError: 190765:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000509aaf6d x1874682943928320/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6327.709013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.771737] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6330.142844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6332.343228] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6332.919098] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6332.950094] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6333.000043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6336.688849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6343.860822] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6352.864527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.869781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.883341] Lustre: Skipped 2 previous similar messages [ 6352.890397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.395225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.410299] Lustre: Skipped 1 previous similar message [ 6354.661361] Lustre: server umount lustre-OST0000 complete [ 6358.504176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.507318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6358.509575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6358.512033] Lustre: Skipped 2 previous similar messages [ 6363.618785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.640119] Lustre: Skipped 2 previous similar messages [ 6364.523111] Lustre: server umount lustre-MDT0000 complete [ 6368.340781] LustreError: 191436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837115 with bad export cookie 2367875298483094449 [ 6368.345285] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6368.353928] LustreError: 191436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6368.768477] Lustre: server umount lustre-MDT0001 complete [ 6381.246265] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6384.250541] Key type lgssc unregistered [ 6384.610714] LNet: 196351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6384.619114] LNetError: 196351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6384.643252] LNet: Removed LNI 192.168.201.107@tcp [ 6385.729671] Key type .llcrypt unregistered [ 6385.731689] Key type ._llcrypt unregistered [ 6412.124063] Key type ._llcrypt registered [ 6412.129686] Key type .llcrypt registered [ 6412.278507] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6413.602495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6413.803925] alg: No test for adler32 (adler32-zlib) [ 6415.203237] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6415.556844] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6417.311240] Key type lgssc registered [ 6418.876674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6428.132643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6428.168696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.818554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.115309] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6440.423847] LustreError: 197764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.443675] LustreError: 197764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6441.340172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.562430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6445.878684] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6452.663470] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6457.671608] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6464.611641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.014316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6470.895284] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6476.334280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6477.874425] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.664230] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6488.588387] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6505.608318] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6522.338915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6522.343108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.361930] LustreError: Skipped 1 previous similar message [ 6522.363700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6522.376538] Lustre: Skipped 1 previous similar message [ 6522.405683] Lustre: Skipped 1 previous similar message [ 6524.269642] Lustre: server umount lustre-OST0000 complete [ 6528.483789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6528.497420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.521994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.583130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.589436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.608164] Lustre: Skipped 1 previous similar message [ 6537.698678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6537.703943] Lustre: Skipped 1 previous similar message [ 6541.793405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.806320] Lustre: Skipped 1 previous similar message [ 6542.303137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6542.514911] Lustre: server umount lustre-MDT0000 complete [ 6546.783108] LustreError: 197746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837294 with bad export cookie 16184428868246181543 [ 6546.790755] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6547.136783] Lustre: server umount lustre-MDT0001 complete [ 6556.289749] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6559.056970] Key type lgssc unregistered [ 6559.334748] LNet: 201656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6559.340706] LNetError: 201656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6560.357964] LNet: Removed LNI 192.168.201.107@tcp [ 6561.382156] Key type .llcrypt unregistered [ 6561.384553] Key type ._llcrypt unregistered [ 6582.649294] Key type ._llcrypt registered [ 6582.652151] Key type .llcrypt registered [ 6582.749239] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6583.824532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6583.854932] alg: No test for adler32 (adler32-zlib) [ 6584.957641] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6585.186460] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6586.831217] Key type lgssc registered [ 6587.715795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6593.671708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6593.689837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.935614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.562275] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6605.282130] LustreError: 202923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.298848] LustreError: 202923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6607.548323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.720452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6611.391175] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6618.230954] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6623.436570] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6630.610383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6631.145474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.297250] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6640.376877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6647.196426] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6652.989690] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6666.971499] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6677.736171] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6682.767526] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6691.818309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6691.837231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.850892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6691.867825] Lustre: Skipped 1 previous similar message [ 6696.642724] Lustre: server umount lustre-OST0000 complete [ 6700.004505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6700.009855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.017575] Lustre: Skipped 1 previous similar message [ 6700.022657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6702.061266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6702.067261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6702.073882] Lustre: Skipped 1 previous similar message [ 6705.590939] Lustre: server umount lustre-MDT0000 complete [ 6707.171735] LustreError: 203650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6707.184636] LustreError: 203650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6709.084885] LustreError: 206444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837456 with bad export cookie 10773138036128972519 [ 6709.087769] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6709.104837] LustreError: 206444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6709.378113] Lustre: server umount lustre-MDT0001 complete [ 6720.420072] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6723.076934] Key type lgssc unregistered [ 6723.531468] LNet: 207898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6723.554260] LNetError: 207898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6724.584690] LNet: Removed LNI 192.168.201.107@tcp [ 6725.517319] Key type .llcrypt unregistered [ 6725.519639] Key type ._llcrypt unregistered [ 6750.335748] Key type ._llcrypt registered [ 6750.337977] Key type .llcrypt registered [ 6750.474968] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6751.402505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6751.423623] alg: No test for adler32 (adler32-zlib) [ 6752.555518] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6752.790153] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6754.489193] Key type lgssc registered [ 6755.595434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6763.586833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6763.612510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.082567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.116850] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6775.278083] LustreError: 209317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.303910] LustreError: 209317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6776.298383] LustreError: 209318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.606511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.090482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6785.275923] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6792.755344] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.616622] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6805.627352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6805.912585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6811.068649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6811.949798] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6822.764305] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6845.931237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.937780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6845.941272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6845.971026] Lustre: Skipped 1 previous similar message [ 6846.006581] Lustre: Skipped 1 previous similar message [ 6848.300266] Lustre: server umount lustre-OST0000 complete [ 6856.176586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.191683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.202568] Lustre: Skipped 1 previous similar message [ 6856.222337] Lustre: Skipped 1 previous similar message [ 6858.998325] Lustre: server umount lustre-MDT0000 complete [ 6861.280536] LustreError: 209317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.292521] LustreError: 209317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6862.603578] LustreError: 209299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837609 with bad export cookie 13794235467875607036 [ 6862.606587] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6862.613374] LustreError: 209299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6862.949360] Lustre: server umount lustre-MDT0001 complete [ 6874.464276] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 6876.825772] Key type lgssc unregistered [ 6877.055791] LNet: 213005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6877.068799] LNetError: 213005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6877.091501] LNet: Removed LNI 192.168.201.107@tcp [ 6877.917950] Key type .llcrypt unregistered [ 6877.921902] Key type ._llcrypt unregistered [ 6898.266459] Key type ._llcrypt registered [ 6898.279984] Key type .llcrypt registered [ 6898.438434] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 6899.833784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6899.857935] alg: No test for adler32 (adler32-zlib) [ 6901.003371] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6901.328371] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 6903.002231] Key type lgssc registered [ 6903.962616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6911.933590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6911.948470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.308856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.980697] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6923.747069] LustreError: 214274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6923.775687] LustreError: 214274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6927.084983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.419525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6931.962911] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6940.134369] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6947.430172] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6955.233328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.733915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.822575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6963.357545] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 6972.416516] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6987.765185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.767281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.793658] Lustre: Skipped 1 previous similar message [ 6992.868686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.890056] Lustre: Skipped 2 previous similar messages [ 6993.949666] Lustre: server umount lustre-OST0000 complete [ 6999.010559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6999.012986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6999.044228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.105167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.108968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.225939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7008.230496] Lustre: Skipped 1 previous similar message [ 7013.344205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7013.726632] Lustre: server umount lustre-MDT0000 complete [ 7018.465171] LustreError: 214269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.485379] LustreError: 214269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7019.077825] LustreError: 217001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837766 with bad export cookie 9806459068160916112 [ 7019.090185] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7019.091172] LustreError: 217001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7019.803566] Lustre: server umount lustre-MDT0001 complete [ 7034.304186] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7037.395820] Key type lgssc unregistered [ 7037.716731] LNet: 217961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7037.725715] LNetError: 217961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.740849] LNet: Removed LNI 192.168.201.107@tcp [ 7038.774346] Key type .llcrypt unregistered [ 7038.776470] Key type ._llcrypt unregistered [ 7061.800774] Key type ._llcrypt registered [ 7061.802145] Key type .llcrypt registered [ 7061.882741] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7063.167704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7063.225045] alg: No test for adler32 (adler32-zlib) [ 7064.293295] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7064.582656] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7066.322289] Key type lgssc registered [ 7067.865431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7076.119413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7076.140661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.643905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.118462] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7087.080418] LustreError: 219231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.099039] LustreError: 219231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7092.198769] LustreError: 219230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.274749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.798926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7098.153222] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7107.267882] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7114.949687] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7124.753546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7125.522138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.894113] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7145.312786] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7147.074109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7156.463236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7156.575696] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7156.844905] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.771226] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7171.578548] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.734378] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7185.881622] Lustre: Permanently deactivating lustre-OST0001 [ 7185.901634] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7195.229265] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7202.811258] Lustre: Permanently reactivating lustre-OST0001 [ 7202.827935] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7202.830956] Lustre: Skipped 2 previous similar messages [ 7204.433695] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7204.455804] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7204.464988] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7204.472195] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7204.502495] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7217.792821] Lustre: server umount lustre-OST0001 complete [ 7220.197967] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7220.200216] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7220.214400] LustreError: Skipped 1 previous similar message [ 7220.228589] LustreError: 221524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.244608] Lustre: Skipped 2 previous similar messages [ 7223.777371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7223.784350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.808356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7224.811062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7224.826388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7225.334134] LustreError: 221055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7225.370587] LustreError: 221055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7228.756213] Lustre: server umount lustre-OST0000 complete [ 7235.557891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.564706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7235.571060] Lustre: Skipped 2 previous similar messages [ 7235.599882] Lustre: Skipped 3 previous similar messages [ 7238.706891] Lustre: server umount lustre-MDT0000 complete [ 7240.686427] LustreError: 219231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.711861] LustreError: 219231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7243.004389] LustreError: 219210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837990 with bad export cookie 8066242918495733580 [ 7243.015634] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.021449] LustreError: 219210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7243.518338] Lustre: server umount lustre-MDT0001 complete [ 7256.586852] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7259.804419] Key type lgssc unregistered [ 7260.140603] LNet: 224304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7260.148527] LNetError: 224304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7260.189060] LNet: Removed LNI 192.168.201.107@tcp [ 7261.318589] Key type .llcrypt unregistered [ 7261.323226] Key type ._llcrypt unregistered [ 7306.772272] Key type ._llcrypt registered [ 7306.774633] Key type .llcrypt registered [ 7306.913691] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7308.610147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7308.623231] alg: No test for adler32 (adler32-zlib) [ 7309.671824] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7309.975501] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7311.672525] Key type lgssc registered [ 7313.231273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7322.256773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7322.268588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.629629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7323.653665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7323.892953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7328.738913] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7338.522761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7338.685886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7338.728944] Lustre: 226508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7339.014789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7343.997895] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7352.895973] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7359.298417] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7367.503355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.662796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7367.972650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.538820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7375.171556] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7383.498962] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7388.106344] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7395.069902] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7404.341910] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7412.401099] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7422.716509] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7440.168808] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7440.178654] Lustre: Skipped 1 previous similar message [ 7475.213506] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7475.220092] Lustre: Skipped 3 previous similar messages [ 7544.499294] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7544.510453] Lustre: Skipped 7 previous similar messages [ 7579.111628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7579.119710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7579.145232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7579.622480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7584.745270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7584.753759] Lustre: Skipped 2 previous similar messages [ 7585.039629] Lustre: server umount lustre-OST0000 complete [ 7589.857293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7589.858050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.880138] Lustre: Skipped 1 previous similar message [ 7589.892678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.904595] Lustre: Skipped 1 previous similar message [ 7594.978990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7600.097231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7600.107059] Lustre: Skipped 2 previous similar messages [ 7604.191216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7604.525868] Lustre: server umount lustre-MDT0000 complete [ 7608.710642] LustreError: 226499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838355 with bad export cookie 12929930549086216453 [ 7608.723941] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7609.579502] Lustre: server umount lustre-MDT0001 complete [ 7623.335962] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7626.320684] Key type lgssc unregistered [ 7626.633199] LNet: 231302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7626.651205] LNetError: 231302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7626.678946] LNet: Removed LNI 192.168.201.107@tcp [ 7627.588430] Key type .llcrypt unregistered [ 7627.591413] Key type ._llcrypt unregistered [ 7656.186054] Key type ._llcrypt registered [ 7656.189827] Key type .llcrypt registered [ 7656.302935] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7657.347972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.431180] alg: No test for adler32 (adler32-zlib) [ 7658.477803] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7658.689477] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7660.335454] Key type lgssc registered [ 7661.915922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7670.347824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7670.360859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.887034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.225654] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7682.021328] LustreError: 232715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.047555] LustreError: 232715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7685.038473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.363704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7690.065692] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7699.782853] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7707.058831] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7715.326578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7715.772746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.875327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7723.679808] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 7731.788220] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7751.469244] Lustre: server umount lustre-OST0000 complete [ 7755.745757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.779294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.787407] Lustre: Skipped 1 previous similar message [ 7760.865428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.875215] Lustre: Skipped 1 previous similar message [ 7765.274984] Lustre: server umount lustre-MDT0000 complete [ 7765.987301] LustreError: 235158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.021259] LustreError: 235158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7770.988461] LustreError: 232695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838518 with bad export cookie 9990182039341404822 [ 7770.998774] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.018332] LustreError: 232695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7781.556255] Lustre: server umount lustre-MDT0001 complete [ 7796.298195] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 7799.761847] Key type lgssc unregistered [ 7800.185494] LNet: 236411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7800.197351] LNetError: 236411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7801.263373] LNet: Removed LNI 192.168.201.107@tcp [ 7802.393702] Key type .llcrypt unregistered [ 7802.399982] Key type ._llcrypt unregistered [ 7829.047791] Key type ._llcrypt registered [ 7829.052149] Key type .llcrypt registered [ 7829.189347] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 7830.397837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.635345] alg: No test for adler32 (adler32-zlib) [ 7831.779288] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7831.984700] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 7833.640562] Key type lgssc registered [ 7835.500922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8107.729602] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8110.578894] Key type lgssc unregistered [ 8110.825982] LNet: 238446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8110.834177] LNetError: 238446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8110.846743] LNet: Removed LNI 192.168.201.107@tcp [ 8111.705524] Key type .llcrypt unregistered [ 8111.713884] Key type ._llcrypt unregistered [ 8139.287788] Key type ._llcrypt registered [ 8139.291689] Key type .llcrypt registered [ 8139.416626] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 8156.642349] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8157.522532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8157.568818] alg: No test for adler32 (adler32-zlib) [ 8158.778731] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 8158.972511] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8160.631196] Key type lgssc registered [ 8161.779906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8197.650736] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7887 sec ========== 09:55:42 (1787838942) [ 8199.544239] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:55:44 (1787838944) === [ 8204.135474] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:55:49 (1787838949) === [ 8252.388848] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8255.517212] Key type lgssc unregistered [ 8255.890523] LNet: 243686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8255.900199] LNetError: 243686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8255.929242] LNet: Removed LNI 192.168.201.107@tcp [ 8256.932551] Key type .llcrypt unregistered [ 8256.940441] Key type ._llcrypt unregistered