[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506883334 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001025] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003020] Switched APIC routing to physical x2apic. [ 0.004023] kvm-guest: setup PV IPIs [ 0.007488] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008041] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009031] pid_max: default: 32768 minimum: 301 [ 0.010242] LSM: Security Framework initializing [ 0.012089] Yama: becoming mindful. [ 0.013074] SELinux: Initializing. [ 0.014128] *** VALIDATE selinux *** [ 0.023444] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028743] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029256] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030159] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031249] *** VALIDATE tmpfs *** [ 0.033138] *** VALIDATE proc *** [ 0.035056] *** VALIDATE cgroup *** [ 0.036019] *** VALIDATE cgroup2 *** [ 0.038157] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040110] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041023] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042044] Spectre V2 : User space: Vulnerable [ 0.043024] Speculative Store Bypass: Vulnerable [ 0.046000] debug: unmapping init [mem 0xffffffffa2459000-0xffffffffa2460fff] [ 0.048341] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049797] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050040] ... version: 2 [ 0.051025] ... bit width: 48 [ 0.052020] ... generic registers: 4 [ 0.053025] ... value mask: 0000ffffffffffff [ 0.054023] ... max period: 00007fffffffffff [ 0.055030] ... fixed-purpose events: 3 [ 0.056018] ... event mask: 000000070000000f [ 0.057551] rcu: Hierarchical SRCU implementation. [ 0.060319] smp: Bringing up secondary CPUs ... [ 0.061887] x86: Booting SMP configuration: [ 0.062075] .... node #0, CPUs: #1 #2 #3 [ 0.066100] smp: Brought up 1 node, 4 CPUs [ 0.068048] smpboot: Max logical packages: 1 [ 0.069020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.238971] node 0 deferred pages initialised in 168ms [ 0.243261] devtmpfs: initialized [ 0.244333] x86/mm: Memory block size: 128MB [ 0.247069] gcov: version magic: 0x41383552 [ 0.250451] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251147] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252508] pinctrl core: initialized pinctrl subsystem [ 0.253359] [ 0.253777] ************************************************************* [ 0.254052] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.255047] ** ** [ 0.256022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257025] ** ** [ 0.258024] ** This means that this kernel is built to expose internal ** [ 0.259027] ** IOMMU data structures, which may compromise security on ** [ 0.260025] ** your system. ** [ 0.261027] ** ** [ 0.262017] ** If you see this message and you are not debugging the ** [ 0.263021] ** kernel, report this immediately to your vendor! ** [ 0.264025] ** ** [ 0.265027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266015] ************************************************************* [ 0.267972] NET: Registered protocol family 16 [ 0.269522] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.272089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.278166] cpuidle: using governor menu [ 0.280222] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.282670] PCI: Using configuration type 1 for base access [ 0.285151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.294157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.300049] cryptd: max_cpu_qlen set to 1000 [ 0.303359] ACPI: Added _OSI(Module Device) [ 0.305067] ACPI: Added _OSI(Processor Device) [ 0.307034] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.309031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314550] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.321329] ACPI: Interpreter enabled [ 0.323104] ACPI: PM: (supports S0 S3 S4 S5) [ 0.324017] ACPI: Using IOAPIC for interrupt routing [ 0.326158] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330448] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.342164] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.346030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.354448] acpiphp: Slot [2] registered [ 0.356151] acpiphp: Slot [5] registered [ 0.357188] acpiphp: Slot [6] registered [ 0.359316] acpiphp: Slot [7] registered [ 0.361242] acpiphp: Slot [8] registered [ 0.362219] acpiphp: Slot [9] registered [ 0.364247] acpiphp: Slot [10] registered [ 0.366309] acpiphp: Slot [3] registered [ 0.368178] acpiphp: Slot [4] registered [ 0.369263] acpiphp: Slot [11] registered [ 0.372153] acpiphp: Slot [12] registered [ 0.373234] acpiphp: Slot [13] registered [ 0.375156] acpiphp: Slot [14] registered [ 0.377154] acpiphp: Slot [15] registered [ 0.378195] acpiphp: Slot [16] registered [ 0.379112] acpiphp: Slot [17] registered [ 0.380158] acpiphp: Slot [18] registered [ 0.381135] acpiphp: Slot [19] registered [ 0.383111] acpiphp: Slot [20] registered [ 0.384121] acpiphp: Slot [21] registered [ 0.385000] acpiphp: Slot [22] registered [ 0.385000] acpiphp: Slot [23] registered [ 0.387138] acpiphp: Slot [24] registered [ 0.388108] acpiphp: Slot [25] registered [ 0.389129] acpiphp: Slot [26] registered [ 0.391180] acpiphp: Slot [27] registered [ 0.393128] acpiphp: Slot [28] registered [ 0.394163] acpiphp: Slot [29] registered [ 0.395054] acpiphp: Slot [30] registered [ 0.396119] acpiphp: Slot [31] registered [ 0.397112] PCI host bridge to bus 0000:00 [ 0.399040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.408037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.410043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.412270] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.415085] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.418361] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.427018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.432971] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.435034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.437022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.439023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.442581] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.445836] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.448065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.452106] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.459029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.474022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.479022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485484] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.495025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.523023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.536474] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.545027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.557028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.586029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.607038] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.631087] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.662025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.677000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.683026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.690019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.706019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716850] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.726315] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.737023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.763020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.772575] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.780020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.790025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.807033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.819144] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.822458] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.824374] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.828677] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831330] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.835161] iommu: Default domain type: Passthrough [ 0.836837] SCSI subsystem initialized [ 0.838380] ACPI: bus type USB registered [ 0.840198] usbcore: registered new interface driver usbfs [ 0.842238] usbcore: registered new interface driver hub [ 0.845155] usbcore: registered new device driver usb [ 0.846369] pps_core: LinuxPPS API ver. 1 registered [ 0.849020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.852130] PTP clock support registered [ 0.855091] EDAC MC: Ver: 3.0.0 [ 0.857251] PCI: Using ACPI for IRQ routing [ 0.859558] NetLabel: Initializing [ 0.860012] NetLabel: domain hash size = 128 [ 0.862014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.864101] NetLabel: unlabeled traffic allowed by default [ 0.865448] vgaarb: loaded [ 0.867420] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.868022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.876000] clocksource: Switched to clocksource kvm-clock [ 0.973361] VFS: Disk quotas dquot_6.6.0 [ 0.975547] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.979336] *** VALIDATE ramfs *** [ 0.980813] *** VALIDATE hugetlbfs *** [ 0.983768] pnp: PnP ACPI init [ 0.986495] pnp: PnP ACPI: found 6 devices [ 1.006056] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.009468] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.011619] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.013937] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.016370] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.018800] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.021671] NET: Registered protocol family 2 [ 1.024489] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.029770] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.033621] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.040335] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.044700] TCP: Hash tables configured (established 65536 bind 65536) [ 1.048598] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.052600] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.055847] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.059426] NET: Registered protocol family 1 [ 1.062564] RPC: Registered named UNIX socket transport module. [ 1.065163] RPC: Registered udp transport module. [ 1.067047] RPC: Registered tcp transport module. [ 1.068889] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.071404] NET: Registered protocol family 44 [ 1.072640] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.073835] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.075644] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.077103] PCI: CLS 0 bytes, default 64 [ 1.078111] Unpacking initramfs... [ 2.554793] debug: unmapping init [mem 0xffff95f4fcc54000-0xffff95f4fffbffff] [ 2.558140] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.560333] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.562664] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.125599] Initialise system trusted keyrings [ 3.127418] Key type blacklist registered [ 3.129331] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.142226] zbud: loaded [ 3.146578] *** VALIDATE nfs *** [ 3.148099] *** VALIDATE nfs4 *** [ 3.150048] pstore: using deflate compression [ 3.154578] Platform Keyring initialized [ 3.276934] NET: Registered protocol family 38 [ 3.278671] Key type asymmetric registered [ 3.280030] Asymmetric key parser 'x509' registered [ 3.281753] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.284656] io scheduler mq-deadline registered [ 3.285839] io scheduler kyber registered [ 3.286813] io scheduler bfq registered [ 3.289251] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.291220] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.293127] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.295261] ACPI: Power Button [PWRF] [ 3.299770] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.305035] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.315600] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.320398] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.332741] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.361205] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.388991] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.394037] Non-volatile memory driver v1.3 [ 3.395460] Linux agpgart interface v0.103 [ 3.429356] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.431391] vda: detected capacity change from 0 to 74760192 [ 3.445248] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.447392] vdb: detected capacity change from 0 to 1073741824 [ 3.460638] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.462855] vdc: detected capacity change from 0 to 2621440000 [ 3.480395] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.484237] vdd: detected capacity change from 0 to 2621440000 [ 3.502847] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.505199] vde: detected capacity change from 0 to 4294967296 [ 3.519658] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.522408] vdf: detected capacity change from 0 to 4294967296 [ 3.532140] libphy: Fixed MDIO Bus: probed [ 3.540604] usbcore: registered new interface driver usbserial_generic [ 3.543489] usbserial: USB Serial support registered for generic [ 3.545834] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.551337] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.553632] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.556811] mousedev: PS/2 mouse device common for all mice [ 3.560025] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.561708] rtc_cmos 00:05: RTC can wake from S4 [ 3.566535] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.566864] rtc_cmos 00:05: registered as rtc0 [ 3.570868] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.572266] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.577991] intel_pstate: CPU model not supported [ 3.581587] hid: raw HID events driver (C) Jiri Kosina [ 3.583890] usbcore: registered new interface driver usbhid [ 3.586140] usbhid: USB HID core driver [ 3.587819] drop_monitor: Initializing network drop monitor service [ 3.590343] Initializing XFRM netlink socket [ 3.592625] NET: Registered protocol family 10 [ 3.595422] Segment Routing with IPv6 [ 3.596809] NET: Registered protocol family 17 [ 3.599415] mpls_gso: MPLS GSO support [ 3.606111] RAS: Correctable Errors collector initialized. [ 3.608225] AVX version of gcm_enc/dec engaged. [ 3.609801] AES CTR mode by8 optimization enabled [ 3.697235] sched_clock: Marking stable (3697194776, 0)->(4714860998, -1017666222) [ 3.701109] registered taskstats version 1 [ 3.703178] Loading compiled-in X.509 certificates [ 3.705296] zswap: loaded using pool lzo/zbud [ 3.733510] Key type big_key registered [ 3.744755] Key type encrypted registered [ 3.746222] ima: No TPM chip found, activating TPM-bypass! [ 3.747712] ima: Allocated hash algorithm: sha1 [ 3.749417] ima: No architecture policies found [ 3.750763] evm: Initialising EVM extended attributes: [ 3.752798] evm: security.selinux [ 3.755623] evm: security.ima [ 3.757090] evm: security.capability [ 3.758504] evm: HMAC attrs: 0x1 [ 3.761349] rtc_cmos 00:05: setting system clock to 2026-08-31 04:57:54 UTC (1788152274) [ 3.768057] debug: unmapping init [mem 0xffffffffa3403000-0xffffffffa35fffff] [ 3.771308] debug: unmapping init [mem 0xffffffffa2182000-0xffffffffa2458fff] [ 3.784109] Write protecting the kernel read-only data: 28672k [ 3.788263] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 3.791914] debug: unmapping init [mem 0xffffffffa1114000-0xffffffffa11fffff] [ 3.834514] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.844056] systemd[1]: Detected virtualization kvm. [ 3.845573] systemd[1]: Detected architecture x86-64. [ 3.847009] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.873268] systemd[1]: No hostname configured. [ 3.875235] systemd[1]: Set hostname to . [ 3.877673] random: systemd: uninitialized urandom read (16 bytes read) [ 3.880391] systemd[1]: Initializing machine ID from random generator. [ 3.946039] random: ln: uninitialized urandom read (6 bytes read) [ 4.043227] random: systemd: uninitialized urandom read (16 bytes read) [ 4.045798] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.051618] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.057976] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.799732] device-mapper: uevent: version 1.0.3 [ 4.801727] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.657562] virtio_net virtio0 ens2: renamed from eth0 [ 5.682700] random: fast init done [ 5.737293] scsi host0: ata_piix [ 5.807790] scsi host1: ata_piix [ 5.809363] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.811332] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.258353] dracut-initqueue[595]: RTNETLINK answers: File exists [ 10.267366] random: crng init done [ 10.268406] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.912531] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.163510] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.436857] SELinux: Disabled at runtime. [ 12.499953] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.508878] systemd[1]: Detected virtualization kvm. [ 12.510865] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.036348] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.040081] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.044823] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.049139] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.053431] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.061481] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.066136] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for det[ 13.155307] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ails. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.560508] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.896357] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.939405] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.066262] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.089658] EDAC sbridge: Ver: 1.1.2 [ 15.577300] Key type dns_resolver registered [ 15.897636] NFS: Registering the id_resolver key type [ 15.900183] Key type id_resolver registered [ 15.902274] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg242-server login: [ 53.688579] hrtimer: interrupt took 3570096 ns [ 77.468111] libcfs: loading out-of-tree module taints kernel. [ 77.553774] Key type ._llcrypt registered [ 77.555932] Key type .llcrypt registered [ 77.636987] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 97.905247] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 100.150821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.166238] alg: No test for adler32 (adler32-zlib) [ 101.509281] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 102.665427] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 104.463648] Key type lgssc registered [ 106.594713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.608338] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.517684] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 192.441894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.491742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.844796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.866101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.967766] Lustre: lustre-MDT0000: new disk, initializing [ 194.099833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.131337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.486934] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.290948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.384561] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.407484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.410935] Lustre: Skipped 1 previous similar message [ 213.513764] Lustre: lustre-MDT0001: new disk, initializing [ 213.585366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.626746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.638470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.354694] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.371513] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 235.533934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.039426] Lustre: lustre-OST0000: new disk, initializing [ 236.048539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.061964] Lustre: 8470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.156199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.268317] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.746824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.760461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.820832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.421289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.740637] Lustre: lustre-OST0001: new disk, initializing [ 260.746204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.758366] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.842988] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.697180] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.888425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.900515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.013480] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 282.685051] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.760599] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 297.220496] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.813267] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing yml_node [ 309.044960] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 311.586416] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 313.835744] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 315.286802] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:03:05 EDT 2026 [ 333.931565] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 335.693937] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.200172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.205681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.207203] Lustre: Skipped 1 previous similar message [ 351.220544] Lustre: Skipped 3 previous similar messages [ 356.323520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.336090] Lustre: Skipped 3 previous similar messages [ 357.374237] Lustre: server umount lustre-MDT0000 complete [ 361.747220] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152632 with bad export cookie 14459852625383683579 [ 361.752488] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.765633] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 366.560813] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.571076] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.592725] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 366.604774] Lustre: Skipped 2 previous similar messages [ 366.616602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.619273] Lustre: Skipped 1 previous similar message [ 371.680448] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.680683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.705928] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 371.708852] Lustre: Skipped 1 previous similar message [ 372.704242] LustreError: 6538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.723729] LustreError: 6538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 376.287329] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.668091] Lustre: server umount lustre-MDT0001 complete [ 381.073714] Lustre: server umount lustre-OST0000 complete [ 385.125646] Lustre: server umount lustre-OST0001 complete [ 391.945108] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 400.212291] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 441.892043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.126626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.152561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.245327] Lustre: lustre-MDT0000: new disk, initializing [ 442.353376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.382407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 445.810651] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 455.425900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.601579] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 455.669260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 455.673268] Lustre: Skipped 1 previous similar message [ 455.718488] Lustre: lustre-MDT0001: new disk, initializing [ 455.820357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.902603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 455.923482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 460.225617] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 471.312233] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 478.352629] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.444924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 485.588495] Lustre: lustre-OST0000: new disk, initializing [ 485.590210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 485.592916] Lustre: 17071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 485.638898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.244972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 487.251890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 487.383095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.544990] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 499.686697] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 504.366100] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.540124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 509.243549] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 509.560931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 512.992885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.998553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.998707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 513.008726] Lustre: Skipped 2 previous similar messages [ 513.025322] Lustre: Skipped 3 previous similar messages [ 518.044725] Lustre: server umount lustre-OST0000 complete [ 522.720844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 522.738248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.767339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.216227] Lustre: server umount lustre-MDT0000 complete [ 532.469337] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.507292] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 535.200794] LustreError: 17070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152805 with bad export cookie 14459852625383685112 [ 535.201778] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.214593] LustreError: 17070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 536.052190] Lustre: server umount lustre-MDT0001 complete [ 541.564453] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:06:50 (1788152810) [ 549.362443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.969475] LustreError: 19335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.173077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.000629] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 559.586819] LustreError: 19335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.605617] LustreError: 19335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 564.373044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.707882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 564.710022] Lustre: Skipped 4 previous similar messages [ 569.044091] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 576.398829] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.101630] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 588.883831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.562794] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 603.688798] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 614.881760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 614.891526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.908991] Lustre: Skipped 1 previous similar message [ 614.934935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.590529] Lustre: server umount lustre-OST0000 complete [ 626.145579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.147208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 626.157662] Lustre: Skipped 2 previous similar messages [ 626.161879] LustreError: Skipped 1 previous similar message [ 628.019822] Lustre: server umount lustre-MDT0000 complete [ 631.265576] LustreError: 19335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.289872] LustreError: 19335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 631.869193] LustreError: 20069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152902 with bad export cookie 14459852625383685994 [ 631.877818] LustreError: 20069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 631.883647] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.164787] Lustre: server umount lustre-MDT0001 complete [ 643.005367] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 645.770735] Key type lgssc unregistered [ 646.039362] LNet: 22866:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.061897] LNetError: 22866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.127915] LNet: Removed LNI 192.168.202.142@tcp [ 647.896259] Key type .llcrypt unregistered [ 647.903343] Key type ._llcrypt unregistered [ 672.927863] Key type ._llcrypt registered [ 672.932143] Key type .llcrypt registered [ 673.049206] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 674.462171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.511799] alg: No test for adler32 (adler32-zlib) [ 675.726690] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 676.101441] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 677.849057] Key type lgssc registered [ 678.830072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 686.565980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 686.579100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.123726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.343191] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 697.319523] LustreError: 24278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.349320] LustreError: 24278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 702.435490] LustreError: 24279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.558234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.061071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 708.288782] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 716.585867] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 723.052963] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 730.536962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.248251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.312526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 737.278688] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 745.370253] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 763.140592] Lustre: server umount lustre-OST0000 complete [ 768.484549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.504852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.600183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.605748] Lustre: Skipped 2 previous similar messages [ 777.864502] Lustre: server umount lustre-MDT0000 complete [ 778.728539] LustreError: 25461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.768473] LustreError: 25461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 779.744044] Lustre: 23590:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153034/real 1788153034] req@00000000303b915f x1875014263534208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788153050 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 779.773988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.784737] Lustre: Skipped 1 previous similar message [ 782.276569] LustreError: 24258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153053 with bad export cookie 13539819363431563545 [ 782.314804] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.821779] Lustre: server umount lustre-MDT0001 complete [ 795.128451] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 798.392757] Key type lgssc unregistered [ 798.708943] LNet: 28257:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.728627] LNetError: 28257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.792976] LNet: Removed LNI 192.168.202.142@tcp [ 801.046261] Key type .llcrypt unregistered [ 801.048348] Key type ._llcrypt unregistered [ 826.057691] Key type ._llcrypt registered [ 826.060138] Key type .llcrypt registered [ 826.207869] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 827.212313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.385094] alg: No test for adler32 (adler32-zlib) [ 828.371960] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 828.567580] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 830.255639] Key type lgssc registered [ 831.694869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 840.113339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 840.153057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.757718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.876904] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 851.936861] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.960243] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 856.061593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.403796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.238611] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 869.757605] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.382470] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 890.988473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.378254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.935296] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 906.253198] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 912.895023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 922.081313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 922.090775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.106332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.111887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.119403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.121091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.595798] Lustre: server umount lustre-OST0000 complete [ 933.344965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.345094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 933.367897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.367906] Lustre: Skipped 1 previous similar message [ 936.307611] Lustre: server umount lustre-MDT0000 complete [ 938.468474] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.495941] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 940.470504] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153211 with bad export cookie 12805523246266800540 [ 940.483499] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.494063] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 941.128685] Lustre: server umount lustre-MDT0001 complete [ 953.111153] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 956.084465] Key type lgssc unregistered [ 956.437581] LNet: 33570:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.441648] LNetError: 33570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.478505] LNet: Removed LNI 192.168.202.142@tcp [ 958.622750] Key type .llcrypt unregistered [ 958.624413] Key type ._llcrypt unregistered [ 982.379359] Key type ._llcrypt registered [ 982.382624] Key type .llcrypt registered [ 982.490945] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 983.257535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.276044] alg: No test for adler32 (adler32-zlib) [ 984.273332] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 984.498556] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 986.192520] Key type lgssc registered [ 987.488326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.275737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.290321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.713719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.564238] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1008.099972] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.129524] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1012.892327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.216149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1013.589653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1020.315655] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1029.397027] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1036.532221] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1044.718112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.115070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.885182] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1060.148445] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1067.649685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1080.802908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1080.811768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.828632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.366523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.381081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.000554] Lustre: server umount lustre-OST0000 complete [ 1093.616399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.627536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.632103] Lustre: Skipped 1 previous similar message [ 1093.643113] Lustre: Skipped 2 previous similar messages [ 1098.154295] Lustre: server umount lustre-MDT0000 complete [ 1098.728028] LustreError: 35909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.741181] LustreError: 35909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1101.956905] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153372 with bad export cookie 554185551526200344 [ 1101.971433] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1101.971859] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1102.315657] Lustre: server umount lustre-MDT0001 complete [ 1115.635435] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1118.625682] Key type lgssc unregistered [ 1118.921766] LNet: 38540:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.933633] LNetError: 38540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.961602] LNet: Removed LNI 192.168.202.142@tcp [ 1119.878143] Key type .llcrypt unregistered [ 1119.879316] Key type ._llcrypt unregistered [ 1145.113957] Key type ._llcrypt registered [ 1145.116805] Key type .llcrypt registered [ 1145.208432] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1146.284647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1146.599791] alg: No test for adler32 (adler32-zlib) [ 1147.737433] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1148.017511] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1149.777585] Key type lgssc registered [ 1150.930142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.526694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.541649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.998538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.981942] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1170.432537] LustreError: 39963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.462538] LustreError: 39963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1175.372406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.524151] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.840537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1180.836871] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1188.217148] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1194.698673] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1201.879271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.204595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.353516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1208.456907] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1217.830879] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1231.138287] Lustre: lustre-OST0000: Not available for connect from 192.168.202.42@tcp (stopping) [ 1231.844672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.847851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.883573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.893331] Lustre: Skipped 1 previous similar message [ 1234.346643] Lustre: server umount lustre-OST0000 complete [ 1256.418176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.420670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.430691] Lustre: Skipped 2 previous similar messages [ 1256.440995] Lustre: Skipped 1 previous similar message [ 1261.507595] Lustre: server umount lustre-MDT0000 complete [ 1261.537955] LustreError: 39959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.552167] LustreError: 39959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1265.933726] LustreError: 39943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153536 with bad export cookie 1730956888212380390 [ 1265.950627] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.478991] Lustre: server umount lustre-MDT0001 complete [ 1286.342840] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1289.899642] Key type lgssc unregistered [ 1290.211253] LNet: 43671:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.223733] LNetError: 43671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.245172] LNet: Removed LNI 192.168.202.142@tcp [ 1291.490496] Key type .llcrypt unregistered [ 1291.494096] Key type ._llcrypt unregistered [ 1316.808907] Key type ._llcrypt registered [ 1316.812407] Key type .llcrypt registered [ 1316.947698] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1318.166280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.227029] alg: No test for adler32 (adler32-zlib) [ 1319.275780] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1319.465434] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1321.161088] Key type lgssc registered [ 1322.415156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.136755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1330.174737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.787084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.001892] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1341.937317] LustreError: 45087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.952551] LustreError: 45087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1347.060305] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.839695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.109229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1352.622884] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1359.905528] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1365.586921] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1371.775534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1372.067570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.337159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1378.734949] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1387.108781] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1393.188510] Lustre: server umount lustre-MDT0000 complete [ 1393.632734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.644975] Lustre: Skipped 1 previous similar message [ 1393.648212] LustreError: 45087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.664661] LustreError: 45087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1396.654361] LustreError: 45066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153667 with bad export cookie 17684179051823423925 [ 1396.655628] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.675290] LustreError: 45066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1398.752562] LustreError: 45081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.753520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.771563] LustreError: 45081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1398.800713] Lustre: Skipped 1 previous similar message [ 1398.807880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1401.760400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1401.769828] Lustre: Skipped 1 previous similar message [ 1403.110294] Lustre: server umount lustre-MDT0001 complete [ 1432.031170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1432.198992] Lustre: server umount lustre-OST0000 complete [ 1449.165923] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1452.576681] Key type lgssc unregistered [ 1452.897742] LNet: 48956:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.900567] LNetError: 48956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.914991] LNet: Removed LNI 192.168.202.142@tcp [ 1453.744280] Key type .llcrypt unregistered [ 1453.749117] Key type ._llcrypt unregistered [ 1481.304518] Key type ._llcrypt registered [ 1481.307342] Key type .llcrypt registered [ 1481.423747] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1482.417057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1482.596117] alg: No test for adler32 (adler32-zlib) [ 1483.657358] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1484.023847] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1485.770040] Key type lgssc registered [ 1487.014870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.770397] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.792229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1511.391776] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1516.511962] LustreError: 50346:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 1543.264100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.798469] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1558.428609] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1725.973234] Lustre: server umount lustre-OST0000 complete [ 1745.963606] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1749.377466] Key type lgssc unregistered [ 1749.664293] LNet: 52022:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.697970] LNetError: 52022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.756914] LNet: Removed LNI 192.168.202.142@tcp [ 1751.808908] Key type .llcrypt unregistered [ 1751.810438] Key type ._llcrypt unregistered [ 1778.390583] Key type ._llcrypt registered [ 1778.392790] Key type .llcrypt registered [ 1778.516876] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1779.806786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.842690] alg: No test for adler32 (adler32-zlib) [ 1781.210222] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1781.426087] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1783.104054] Key type lgssc registered [ 1784.565467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.379882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.391772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.953915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.806040] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1805.281417] LustreError: 53438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.315863] LustreError: 53438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1810.418847] LustreError: 53437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.210408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.794668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.819198] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1825.760839] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.507949] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1840.196108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.548917] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1840.664166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.754522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1847.888406] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1856.612145] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1860.071870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.076472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1860.088265] Lustre: Skipped 1 previous similar message [ 1860.096632] Lustre: Skipped 1 previous similar message [ 1865.148767] Lustre: server umount lustre-OST0000 complete [ 1870.306547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.312499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.342034] Lustre: Skipped 1 previous similar message [ 1870.365860] Lustre: Skipped 1 previous similar message [ 1875.424635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.134884] Lustre: server umount lustre-MDT0000 complete [ 1879.522807] LustreError: 53438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.538761] LustreError: 53438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1880.686342] LustreError: 54171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154151 with bad export cookie 718207508944900633 [ 1880.706797] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.337804] Lustre: server umount lustre-MDT0001 complete [ 1898.317752] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1901.163819] Key type lgssc unregistered [ 1901.634854] LNet: 56980:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.647646] LNetError: 56980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.669429] LNet: Removed LNI 192.168.202.142@tcp [ 1902.905688] Key type .llcrypt unregistered [ 1902.907625] Key type ._llcrypt unregistered [ 1928.554958] Key type ._llcrypt registered [ 1928.557237] Key type .llcrypt registered [ 1928.666118] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1929.561845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.570569] alg: No test for adler32 (adler32-zlib) [ 1930.764660] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1931.301107] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1933.231223] Key type lgssc registered [ 1934.606127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.605976] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.629302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.287415] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1965.407698] LustreError: 58366:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 1992.161831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.164798] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2008.369071] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.122888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.455497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.521251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2022.661730] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2022.881417] LustreError: 58392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.889964] LustreError: 58392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2027.006487] LustreError: 59403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.031827] LustreError: 59403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2032.064401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.114465] LustreError: 59404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.132525] LustreError: 59404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2032.763834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.860054] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2048.362586] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.834511] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2058.230396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.241974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.250143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2063.335686] LustreError: 58383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.710434] Lustre: server umount lustre-OST0000 complete [ 2088.936362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.945038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2088.953032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.953041] Lustre: Skipped 1 previous similar message [ 2088.966611] Lustre: Skipped 2 previous similar messages [ 2090.032399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.043042] Lustre: Skipped 2 previous similar messages [ 2094.917341] Lustre: server umount lustre-MDT0000 complete [ 2095.079865] LustreError: 59402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.095829] LustreError: 59402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2099.079931] LustreError: 58372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154369 with bad export cookie 10750486986399857553 [ 2099.081282] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.089034] LustreError: 58372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.644221] Lustre: server umount lustre-MDT0001 complete [ 2119.612857] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2123.152850] Key type lgssc unregistered [ 2123.536654] LNet: 62114:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.545515] LNetError: 62114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.582798] LNet: Removed LNI 192.168.202.142@tcp [ 2125.831960] Key type .llcrypt unregistered [ 2125.833904] Key type ._llcrypt unregistered [ 2156.526361] Key type ._llcrypt registered [ 2156.531610] Key type .llcrypt registered [ 2156.654243] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2158.215395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2158.365591] alg: No test for adler32 (adler32-zlib) [ 2159.705199] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2160.057661] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2161.847381] Key type lgssc registered [ 2163.334800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.348027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2173.359360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.877204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.957536] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2185.190223] LustreError: 63528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.229032] LustreError: 63528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2190.317245] LustreError: 63529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.069090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.363420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.787577] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2204.542241] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.468731] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2218.295879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2218.640689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.700929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2225.435201] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2232.021666] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2249.534288] Lustre: server umount lustre-OST0000 complete [ 2257.898422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.904363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.916984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.923942] Lustre: Skipped 1 previous similar message [ 2263.010117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.016356] Lustre: Skipped 1 previous similar message [ 2264.163366] Lustre: server umount lustre-MDT0000 complete [ 2268.133134] LustreError: 63523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.160578] LustreError: 63523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2268.804358] LustreError: 63508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154539 with bad export cookie 2210121324070742370 [ 2268.823752] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2278.713584] Lustre: server umount lustre-MDT0001 complete [ 2292.900323] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2296.400274] Key type lgssc unregistered [ 2296.826661] LNet: 67083:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.836725] LNetError: 67083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.890514] LNet: Removed LNI 192.168.202.142@tcp [ 2298.409285] Key type .llcrypt unregistered [ 2298.410567] Key type ._llcrypt unregistered [ 2329.744783] Key type ._llcrypt registered [ 2329.748556] Key type .llcrypt registered [ 2329.825177] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2331.286151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.330709] alg: No test for adler32 (adler32-zlib) [ 2332.387756] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2332.738429] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2334.495349] Key type lgssc registered [ 2335.575805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2344.968564] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:36:54 (1788154614) [ 2352.145510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2352.190206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.726583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.276382] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2363.879240] LustreError: 68787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.892473] LustreError: 68787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2368.995097] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.395330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.045559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.879106] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2383.911386] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2390.748535] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2398.492967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.695368] LustreError: 70609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.855849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.365183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2406.263038] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2416.010398] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.908634] Lustre: Failing over lustre-MDT0000 [ 2423.199762] Lustre: server umount lustre-MDT0000 complete [ 2424.102650] LustreError: 68782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.852241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.869975] Lustre: Skipped 2 previous similar messages [ 2429.187027] LustreError: 68782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.206077] LustreError: 68782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2433.720759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.916951] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.181882] Lustre: *** cfs_fail_loc=135, val=0*** [ 2434.308566] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.42@tcp (not set up) [ 2434.365158] LustreError: 71879:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2434.378107] LustreError: 71879:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.142@tcp: cfg command failed: rc = -2 [ 2434.396827] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2434.396827] [ 2434.409635] LustreError: MGC192.168.202.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2434.433446] LustreError: 71867:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2434.452031] LustreError: 71867:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 2434.469853] LustreError: 71867:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2434.492188] LustreError: 71867:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2434.554493] Lustre: server umount lustre-MDT0000 complete [ 2434.557975] LustreError: 71867:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2439.422714] LustreError: 68783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.437960] LustreError: 68783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2441.952404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.134666] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2442.576824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.650661] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2444.543671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2447.195818] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2447.872447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2447.901890] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2447.959333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2457.570322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.586403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.616158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.206442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.210841] Lustre: Skipped 3 previous similar messages [ 2468.325135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.338416] Lustre: Skipped 1 previous similar message [ 2469.345173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2469.585385] Lustre: server umount lustre-OST0000 complete [ 2478.562159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2478.567644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.625358] Lustre: Skipped 1 previous similar message [ 2478.631536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.634190] Lustre: Skipped 1 previous similar message [ 2480.126661] Lustre: server umount lustre-MDT0000 complete [ 2483.688265] LustreError: 68784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.714461] LustreError: 68784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2484.655266] LustreError: 71370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154755 with bad export cookie 15265776585678847523 [ 2484.661694] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.663559] LustreError: 71370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2485.236572] Lustre: server umount lustre-MDT0001 complete [ 2498.991151] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2502.173825] Key type lgssc unregistered [ 2502.523447] LNet: 73660:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.539920] LNetError: 73660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.564179] LNet: Removed LNI 192.168.202.142@tcp [ 2503.667427] Key type .llcrypt unregistered [ 2503.669499] Key type ._llcrypt unregistered [ 2530.269501] Key type ._llcrypt registered [ 2530.271779] Key type .llcrypt registered [ 2530.372505] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2531.427784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2531.475483] alg: No test for adler32 (adler32-zlib) [ 2532.576334] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2532.854363] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2534.567337] Key type lgssc registered [ 2536.206106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2547.484705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2547.503898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.186531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.112959] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2558.441972] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.462423] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2563.565516] LustreError: 75073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.932654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.451738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2570.874296] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2580.145332] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.556329] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2593.629705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.876410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.849947] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2608.552274] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2618.452991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2624.442577] Lustre: Failing over lustre-MDT0000 [ 2624.787133] LustreError: 75068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.817637] Lustre: server umount lustre-MDT0000 complete [ 2627.040167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2627.043303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.074298] LustreError: 75802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.582929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.707224] LustreError: 77162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.754070] LustreError: 77162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2636.340158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.577641] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.870744] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2637.029800] LustreError: 78181:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2637.038946] LustreError: 78181:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.142@tcp: cfg command failed: rc = -9 [ 2637.046327] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2637.046327] [ 2637.058619] LustreError: MGC192.168.202.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2637.067869] LustreError: 78169:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2637.082191] LustreError: 78169:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -9 [ 2637.098623] LustreError: 78169:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2637.108142] LustreError: 78169:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2637.124052] Lustre: server umount lustre-MDT0000 complete [ 2637.126220] LustreError: 78169:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2643.942441] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.969892] LustreError: 75072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2645.017088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.229139] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.507275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.558599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2646.269218] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2650.332898] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2650.640022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2650.720199] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2650.798925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2660.837281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.851940] Lustre: Skipped 2 previous similar messages [ 2660.852466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.971419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.993372] Lustre: Skipped 2 previous similar messages [ 2666.922121] Lustre: server umount lustre-OST0000 complete [ 2671.079143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.100501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.104639] Lustre: Skipped 1 previous similar message [ 2676.193150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.199896] Lustre: Skipped 1 previous similar message [ 2677.470346] Lustre: server umount lustre-MDT0000 complete [ 2681.321637] LustreError: 75069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.348092] LustreError: 75069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2683.205392] LustreError: 75054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154953 with bad export cookie 3365561291096418519 [ 2683.210992] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.236612] LustreError: 75054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2683.743707] Lustre: server umount lustre-MDT0001 complete [ 2695.807628] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2700.342000] Key type lgssc unregistered [ 2700.839791] LNet: 79964:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.859637] LNetError: 79964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.883632] LNet: Removed LNI 192.168.202.142@tcp [ 2702.629976] Key type .llcrypt unregistered [ 2702.634750] Key type ._llcrypt unregistered [ 2722.123865] Key type ._llcrypt registered [ 2722.126219] Key type .llcrypt registered [ 2722.281806] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2722.727506] Key type .llcrypt unregistered [ 2722.731243] Key type ._llcrypt unregistered [ 2752.362368] Key type ._llcrypt registered [ 2752.372263] Key type .llcrypt registered [ 2752.571868] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2754.268307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2754.281168] alg: No test for adler32 (adler32-zlib) [ 2755.342694] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2755.611385] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2757.287288] Key type lgssc registered [ 2758.529343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.385749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.420980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.055887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.165596] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2778.604080] LustreError: 81870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.633148] LustreError: 81870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2783.380099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.723507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.497588] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2797.834284] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2805.089754] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2814.020458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.629564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.305858] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2831.887482] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2839.236836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2849.764353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.794163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.802425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.822681] Lustre: Skipped 1 previous similar message [ 2851.799993] Lustre: server umount lustre-OST0000 complete [ 2860.002567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.034118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.040376] Lustre: Skipped 2 previous similar messages [ 2860.069912] Lustre: Skipped 1 previous similar message [ 2864.543516] Lustre: server umount lustre-MDT0000 complete [ 2865.126393] LustreError: 84363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.136676] LustreError: 84363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2867.964592] LustreError: 81850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155138 with bad export cookie 6055808882777179994 [ 2867.979927] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2867.984870] LustreError: 81850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2868.321493] Lustre: server umount lustre-MDT0001 complete [ 2878.820168] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2881.718533] Key type lgssc unregistered [ 2882.058541] LNet: 85426:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2882.066268] LNetError: 85426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.082483] LNet: Removed LNI 192.168.202.142@tcp [ 2883.580560] Key type .llcrypt unregistered [ 2883.586291] Key type ._llcrypt unregistered [ 2907.947967] Key type ._llcrypt registered [ 2907.952576] Key type .llcrypt registered [ 2908.116909] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2909.415672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2909.558798] alg: No test for adler32 (adler32-zlib) [ 2910.695632] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2910.985771] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2912.751394] Key type lgssc registered [ 2914.303159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2922.974160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2922.995082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.477030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.299533] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2934.753733] LustreError: 86855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.784911] LustreError: 86855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2939.487539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.956778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.528312] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2955.171828] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2962.363882] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2971.128726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.556154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.251835] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2987.561534] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2995.121690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3005.410879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.412706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3005.461843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3005.488443] Lustre: Skipped 1 previous similar message [ 3010.541232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.548890] Lustre: Skipped 1 previous similar message [ 3010.985301] Lustre: server umount lustre-OST0000 complete [ 3015.658156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.679091] Lustre: Skipped 1 previous similar message [ 3015.689260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.771196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.781222] Lustre: Skipped 2 previous similar messages [ 3022.148570] Lustre: server umount lustre-MDT0000 complete [ 3025.890471] LustreError: 89152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.912251] LustreError: 89152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3026.914683] LustreError: 86834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155297 with bad export cookie 14825930670307957001 [ 3026.916801] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.922481] LustreError: 86834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3027.237824] Lustre: server umount lustre-MDT0001 complete [ 3041.800283] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 3045.234823] Key type lgssc unregistered [ 3045.510868] LNet: 90413:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3045.528198] LNetError: 90413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3045.567901] LNet: Removed LNI 192.168.202.142@tcp [ 3046.848678] Key type .llcrypt unregistered [ 3046.850304] Key type ._llcrypt unregistered [ 3072.892841] Key type ._llcrypt registered [ 3072.895358] Key type .llcrypt registered [ 3073.055675] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3074.432203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3074.643511] alg: No test for adler32 (adler32-zlib) [ 3075.923406] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3076.239397] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 3077.963146] Key type lgssc registered [ 3080.130851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3091.812490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3091.861891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.383399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.379043] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3103.739259] LustreError: 91828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.750603] LustreError: 91828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3108.836544] LustreError: 91827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.255127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.662759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3114.068236] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3121.911407] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3128.165587] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3135.819869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.237339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.306742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3143.009889] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3150.373881] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3167.205086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3167.213030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.236955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.921534] Lustre: server umount lustre-OST0000 complete [ 3175.394357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.399198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.407061] Lustre: Skipped 1 previous similar message [ 3179.125971] Lustre: server umount lustre-MDT0000 complete [ 3180.517413] LustreError: 92556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.541949] LustreError: 92556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3182.819181] LustreError: 91809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155453 with bad export cookie 11305931907194229504 [ 3182.833950] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.458157] Lustre: server umount lustre-MDT0001 complete [ 3203.691038] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 3206.735282] Key type lgssc unregistered [ 3207.206378] LNet: 95389:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.213368] LNetError: 95389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.229758] LNet: Removed LNI 192.168.202.142@tcp [ 3208.176989] Key type .llcrypt unregistered [ 3208.179493] Key type ._llcrypt unregistered [ 3233.956638] Key type ._llcrypt registered [ 3233.958902] Key type .llcrypt registered [ 3234.079211] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3235.390879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3235.424154] alg: No test for adler32 (adler32-zlib) [ 3236.403710] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3236.598191] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 3238.295200] Key type lgssc registered [ 3239.533606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3250.038451] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3250.087648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3265.823460] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3270.943587] LustreError: 96777:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 3297.567469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.362703] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3313.783674] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3320.015190] Lustre: server umount lustre-OST0000 complete [ 3329.399585] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:53:18 (1788155598) [ 3335.985217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.446372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.463492] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3341.793940] LustreError: 98416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.809943] LustreError: 98416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3346.916623] LustreError: 98417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.212082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.519106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3355.668965] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3363.835151] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3370.219429] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3378.000893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.321392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.454041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3383.914203] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3391.077366] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.306737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.308702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.320869] Lustre: Skipped 1 previous similar message [ 3410.188930] Lustre: server umount lustre-OST0000 complete [ 3416.546312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.565414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.577725] Lustre: Skipped 2 previous similar messages [ 3421.667369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.680081] Lustre: Skipped 1 previous similar message [ 3426.784514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.798349] Lustre: Skipped 1 previous similar message [ 3428.319693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3428.643354] Lustre: server umount lustre-MDT0000 complete [ 3431.914448] LustreError: 98417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.946724] LustreError: 98417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3432.492784] LustreError: 99151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155703 with bad export cookie 3142646344755256608 [ 3432.508375] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.516864] LustreError: 99151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3432.874209] Lustre: server umount lustre-MDT0001 complete [ 3446.310501] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 3451.456568] Key type lgssc unregistered [ 3451.966654] LNet: 102113:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3451.988650] LNetError: 102113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3452.019555] LNet: Removed LNI 192.168.202.142@tcp [ 3453.640696] Key type .llcrypt unregistered [ 3453.642686] Key type ._llcrypt unregistered [ 3501.264581] Key type ._llcrypt registered [ 3501.266952] Key type .llcrypt registered [ 3501.362780] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3502.972382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3503.008733] alg: No test for adler32 (adler32-zlib) [ 3504.385588] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3504.690910] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 3506.423399] Key type lgssc registered [ 3508.092205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3518.044085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3518.065672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.763747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.130860] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3530.212831] LustreError: 104301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.242587] LustreError: 104301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3535.338398] LustreError: 104302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.447282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.958899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3540.764738] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3550.354383] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3557.188142] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.570195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.897162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.485424] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3583.725490] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3589.446559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3599.843967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.858270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3599.860203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3599.869680] Lustre: Skipped 1 previous similar message [ 3601.533038] Lustre: server umount lustre-OST0000 complete [ 3607.520597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3607.527806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.534552] Lustre: Skipped 1 previous similar message [ 3607.538604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.081048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.083484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.859154] Lustre: server umount lustre-MDT0000 complete [ 3615.206557] LustreError: 104301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.258579] LustreError: 104301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3617.452896] LustreError: 104280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155888 with bad export cookie 11230529168798615476 [ 3617.461565] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3617.472065] LustreError: 104280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3618.162532] Lustre: server umount lustre-MDT0001 complete [ 3632.639304] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 3635.651843] Key type lgssc unregistered [ 3635.930332] LNet: 107860:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3635.939223] LNetError: 107860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3636.966375] LNet: Removed LNI 192.168.202.142@tcp [ 3638.002755] Key type .llcrypt unregistered [ 3638.007763] Key type ._llcrypt unregistered [ 3658.711472] Key type ._llcrypt registered [ 3658.712876] Key type .llcrypt registered [ 3658.881414] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3659.992413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3660.301239] alg: No test for adler32 (adler32-zlib) [ 3661.397688] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3661.700458] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 3663.431244] Key type lgssc registered [ 3664.390240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3672.568378] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3672.600918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.225129] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3693.345109] LustreError: 109103:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 3720.031189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.494872] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3739.790297] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3747.813301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.099210] LustreError: MGC192.168.202.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3748.108873] LustreError: 110126:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3748.118349] LustreError: 110126:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 3748.128281] LustreError: 110126:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 3748.231228] Lustre: server umount lustre-MDT0000 complete [ 3748.233922] LustreError: 110126:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3765.854706] Lustre: server umount lustre-OST0000 complete [ 3777.289667] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 3789.740312] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3836.970795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.330442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3837.365638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3837.472923] Lustre: lustre-MDT0000: new disk, initializing [ 3837.591567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.612622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3842.251743] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3853.119383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.261603] Lustre: 112933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3853.335359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3853.341071] Lustre: Skipped 1 previous similar message [ 3853.472175] Lustre: lustre-MDT0001: new disk, initializing [ 3853.569876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.635255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3853.649109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3858.535155] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3869.698742] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3875.922985] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3884.045702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.374221] Lustre: lustre-OST0000: new disk, initializing [ 3884.377859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3884.384190] Lustre: 114902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3884.509198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.276956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3891.293145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3891.510046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3891.720488] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3903.526132] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3908.411311] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3908.663400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3914.093790] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3914.446918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3917.282899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3917.292908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.307189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.312897] Lustre: Skipped 1 previous similar message [ 3922.405232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.426548] Lustre: server umount lustre-OST0000 complete [ 3930.598478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3930.608263] LustreError: Skipped 1 previous similar message [ 3930.612312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.626297] Lustre: Skipped 1 previous similar message [ 3930.632218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.639592] Lustre: Skipped 1 previous similar message [ 3932.645819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.647095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.761611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3937.772484] Lustre: Skipped 2 previous similar messages [ 3942.367298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3943.082584] Lustre: server umount lustre-MDT0000 complete [ 3948.004369] LustreError: 112945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.031474] LustreError: 112945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3949.414028] LustreError: 113728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156220 with bad export cookie 1435818332858955540 [ 3949.438136] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3949.455545] LustreError: 113728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3950.444827] Lustre: server umount lustre-MDT0001 complete [ 3961.495883] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:03:51 (1788156231) [ 3965.444525] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3984.429647] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 3996.583365] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 4035.406321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.772764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4035.807844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4035.892501] Lustre: lustre-MDT0000: new disk, initializing [ 4035.995905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.024466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4040.756605] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4051.615548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.706676] Lustre: 119807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4051.752631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4051.759924] Lustre: Skipped 1 previous similar message [ 4051.885423] Lustre: lustre-MDT0001: new disk, initializing [ 4051.962560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4052.020533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4052.030768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4057.009847] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4067.742735] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.258696] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4081.423572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.715177] Lustre: lustre-OST0000: new disk, initializing [ 4081.718464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4081.725776] Lustre: 121773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4081.865045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.630317] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4090.493053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4090.513544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4090.643033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4100.369900] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.412426] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.609497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4110.435961] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4110.736778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4116.449452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.451467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.475257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.199860] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4127.417910] Lustre: server umount lustre-OST0000 complete [ 4131.828993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4131.836588] Lustre: Skipped 2 previous similar messages [ 4136.933660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.937963] Lustre: Skipped 8 previous similar messages [ 4146.143138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4146.455948] Lustre: server umount lustre-MDT0000 complete [ 4147.169286] LustreError: 121222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.190755] LustreError: 121222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4150.581624] LustreError: 119799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156421 with bad export cookie 1435818332858956422 [ 4150.582934] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.593150] LustreError: 119799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4151.072806] Lustre: server umount lustre-MDT0001 complete [ 4160.640157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.017346] LustreError: 123994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.105853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.282492] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4166.130985] LustreError: 123995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.273492] LustreError: 123994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.728287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.855485] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4188.886551] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4196.531133] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4205.549592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4215.095485] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4225.207978] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4233.389499] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4237.792544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4237.796718] LustreError: Skipped 1 previous similar message [ 4237.798674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.809788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4237.813812] Lustre: Skipped 2 previous similar messages [ 4244.166746] Lustre: server umount lustre-OST0000 complete [ 4255.845567] Lustre: server umount lustre-MDT0000 complete [ 4259.297095] LustreError: 123994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.320586] LustreError: 123994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4260.385195] LustreError: 126873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156531 with bad export cookie 1435818332858957304 [ 4260.391434] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.405897] LustreError: 126873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4260.970732] Lustre: server umount lustre-MDT0001 complete [ 4274.327518] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 4278.047851] Key type lgssc unregistered [ 4278.400076] LNet: 127629:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4278.407593] LNetError: 127629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4278.426687] LNet: Removed LNI 192.168.202.142@tcp [ 4279.851450] Key type .llcrypt unregistered [ 4279.854303] Key type ._llcrypt unregistered [ 4301.639475] Key type ._llcrypt registered [ 4301.646135] Key type .llcrypt registered [ 4301.726378] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 4318.686656] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 4320.142071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4320.334028] alg: No test for adler32 (adler32-zlib) [ 4321.523274] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4321.809921] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 4323.535214] Key type lgssc registered [ 4324.642633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4365.133817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.227431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.652675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4366.710543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4366.839347] Lustre: lustre-MDT0000: new disk, initializing [ 4366.935543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.953447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4372.708102] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4384.687225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.855900] Lustre: 130788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4384.930616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4384.941544] Lustre: Skipped 1 previous similar message [ 4385.153852] Lustre: lustre-MDT0001: new disk, initializing [ 4385.240958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4385.292654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4385.305227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4391.545866] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4404.895876] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4411.367799] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.910965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.310608] Lustre: lustre-OST0000: new disk, initializing [ 4420.313786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4420.335299] Lustre: 132757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4420.459894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.809911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4426.823660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4426.982192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4429.180044] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4440.108129] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4446.916991] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4447.436450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4455.013778] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4455.466810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4463.102425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.119036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4463.120049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4463.122509] Lustre: Skipped 1 previous similar message [ 4463.152646] Lustre: Skipped 1 previous similar message [ 4464.804727] Lustre: server umount lustre-OST0000 complete [ 4472.296135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4472.305549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.338129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4473.319682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.364631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4473.374737] Lustre: Skipped 1 previous similar message [ 4476.230862] Lustre: server umount lustre-MDT0000 complete [ 4481.296222] LustreError: 130779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156752 with bad export cookie 12888232638361348464 [ 4481.297686] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4481.317913] LustreError: 130779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4481.744564] Lustre: server umount lustre-MDT0001 complete [ 4493.472398] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:12:43 (1788156763) [ 4501.295895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.655076] LustreError: 135171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.672183] LustreError: 135171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4501.769809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.110144] LustreError: 135172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.612518] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4512.225508] LustreError: 135171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.361135] LustreError: 135172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.665876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.055328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4523.416204] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4532.947635] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4539.056617] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4542.306179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.323388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4542.329023] Lustre: Skipped 1 previous similar message [ 4547.557620] LustreError: 135172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.599429] Lustre: server umount lustre-MDT0000 complete [ 4551.852136] LustreError: 135153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156822 with bad export cookie 12888232638361349353 [ 4551.875046] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.875794] LustreError: 135153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4552.393651] Lustre: server umount lustre-MDT0001 complete [ 4561.785832] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:13:51 (1788156831) [ 4568.458273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.099328] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4589.215390] LustreError: 137584:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 4614.881458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.063305] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4630.587757] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4632.930197] Lustre: server umount lustre-OST0000 complete [ 4641.989515] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:15:11 (1788156911) [ 4650.125742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.065924] LustreError: 139032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.084717] LustreError: 139032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4651.176379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.750977] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4668.223088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.411609] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4684.866606] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4691.635858] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4700.048468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4700.703342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.715553] Lustre: Skipped 1 previous similar message [ 4708.798561] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4720.265860] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4731.888701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.901148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.911958] Lustre: Skipped 3 previous similar messages [ 4731.943138] Lustre: Skipped 2 previous similar messages [ 4734.009595] Lustre: server umount lustre-MDT0000 complete [ 4737.075143] LustreError: 139762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.120415] LustreError: 139762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4739.215431] LustreError: 139014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157009 with bad export cookie 12888232638361350018 [ 4739.225318] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4739.235841] LustreError: 139014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4739.721193] Lustre: server umount lustre-MDT0001 complete [ 4754.168550] Lustre: server umount lustre-OST0000 complete [ 4764.142509] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:17:13 (1788157033) [ 4771.955581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.384527] LustreError: 142688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.396332] LustreError: 142688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4772.470137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.915196] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4788.860627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.102190] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4804.257810] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4810.654418] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4818.396507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4826.528637] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4834.276993] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4839.430275] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4842.432092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4843.004888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4849.518687] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4849.790913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4852.704207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4852.713048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.731831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4852.740470] Lustre: Skipped 1 previous similar message [ 4857.889198] Lustre: server umount lustre-OST0000 complete [ 4868.858324] Lustre: server umount lustre-MDT0000 complete [ 4873.188211] LustreError: 142688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.224890] LustreError: 142688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4875.114863] LustreError: 142670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157145 with bad export cookie 12888232638361351089 [ 4875.143221] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4875.150241] LustreError: 142670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4875.695868] Lustre: server umount lustre-MDT0001 complete [ 4886.304794] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:19:14 (1788157154) [ 4894.711062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4910.303843] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4915.424594] LustreError: 146870:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 4941.151261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.158604] Lustre: Skipped 2 previous similar messages [ 4947.572358] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4956.293584] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4966.134603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4967.001544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4972.460263] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4982.008510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4988.481987] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4998.910353] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5006.393766] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5013.848486] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5014.334925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5020.455797] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5020.682664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5023.716270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.728876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5023.731253] Lustre: Skipped 4 previous similar messages [ 5023.759951] Lustre: Skipped 5 previous similar messages [ 5028.320747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5030.091986] Lustre: server umount lustre-MDT0000 complete [ 5033.972101] LustreError: 148645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.010974] LustreError: 148645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5035.903249] LustreError: 146879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157306 with bad export cookie 12888232638361352111 [ 5035.904887] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.906393] LustreError: 146879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5036.125233] Lustre: server umount lustre-MDT0001 complete [ 5048.532845] Lustre: server umount lustre-OST0000 complete [ 5058.465662] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:22:07 (1788157327) [ 5067.330783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5082.977320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5088.099872] LustreError: 151071:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 5121.434707] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5130.570364] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5138.843957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.478091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5144.919632] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5155.213236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.702520] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5168.604544] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5175.447974] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5182.868767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.014216] Lustre: lustre-OST0001: new disk, initializing [ 5183.016956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5183.023399] Lustre: 153945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5188.648379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5188.657270] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5188.799711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5188.915343] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5201.627477] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5209.626698] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5210.060226] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5218.601579] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5219.272694] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5224.957692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.976032] Lustre: Skipped 1 previous similar message [ 5224.982699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5225.000165] Lustre: Skipped 4 previous similar messages [ 5228.757526] Lustre: server umount lustre-OST0000 complete [ 5234.660834] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.220356] Lustre: server umount lustre-OST0001 complete [ 5250.972834] Lustre: server umount lustre-MDT0000 complete [ 5255.227239] LustreError: 153944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157525 with bad export cookie 12888232638361353063 [ 5255.229430] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5255.237662] LustreError: 153944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5255.695327] Lustre: server umount lustre-MDT0001 complete [ 5284.670663] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:25:53 (1788157553) [ 5286.927178] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5289.247118] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5291.531320] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:26:00 (1788157560) [ 5299.513703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5299.844920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5299.893310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5300.105761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5300.115797] Lustre: Skipped 6 previous similar messages [ 5305.449669] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5315.858377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.928223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5315.951440] Lustre: 157509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5320.486971] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5328.342393] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.999873] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5345.253077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.542567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5352.201328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5353.951980] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5363.756613] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.330554] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5369.518657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5374.337948] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5374.625364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5378.022264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5378.029658] LustreError: Skipped 1 previous similar message [ 5378.032394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.041648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.046672] Lustre: Skipped 6 previous similar messages [ 5378.064742] Lustre: Skipped 8 previous similar messages [ 5391.328653] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5391.499274] Lustre: server umount lustre-OST0000 complete [ 5417.226985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.688584] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.705681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.707980] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5424.549393] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5432.014593] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5439.475534] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5439.695423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5445.295826] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5445.675689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5452.386210] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff949047b27800.ost_server_uuid 50 [ 5454.308337] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff949047b27800.ost_server_uuid in FULL state after 0 sec [ 5454.517676] LustreError: 162491:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.42@tcp) returned error from blocking AST (req@00000000f00e550e x1875018086902784 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f4082545/0xb2dc334cc01fc4c1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.42@tcp remote: 0x57894307d9db44fa expref: 5 pid: 157516 timeout: 5554 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5454.541614] LustreError: lustre-MDT0000: A client on nid 192.168.202.42@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5454.558287] LustreError: 157506:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.42@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f4082545/0xb2dc334cc01fc4c1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.42@tcp remote: 0x57894307d9db44fa expref: 6 pid: 157516 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5459.951742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5459.958887] LustreError: Skipped 1 previous similar message [ 5464.963893] Lustre: server umount lustre-OST0000 complete [ 5476.026471] Lustre: server umount lustre-MDT0000 complete [ 5479.404286] LustreError: 157516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.442350] LustreError: 157516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5481.124669] LustreError: 159382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157751 with bad export cookie 12888232638361354246 [ 5481.128321] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.135088] LustreError: 159382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5487.633288] Lustre: server umount lustre-MDT0001 complete [ 5501.408244] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 5504.924600] Key type lgssc unregistered [ 5505.324283] LNet: 163630:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5505.341439] LNetError: 163630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5505.362460] LNet: Removed LNI 192.168.202.142@tcp [ 5506.333353] Key type .llcrypt unregistered [ 5506.336819] Key type ._llcrypt unregistered [ 5535.497860] Key type ._llcrypt registered [ 5535.505629] Key type .llcrypt registered [ 5535.641099] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 5537.211773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5537.265684] alg: No test for adler32 (adler32-zlib) [ 5538.583766] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5539.085973] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 5540.959420] Key type lgssc registered [ 5542.370767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5552.976852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5553.045195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.252987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5562.066844] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5564.391601] LustreError: 165044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.420167] LustreError: 165044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5569.507708] LustreError: 165045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.224316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.646392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5579.335855] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5588.990531] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5596.768364] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5605.666210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.158594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5612.660372] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5621.071977] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5624.653904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5631.353983] Lustre: Failing over lustre-MDT0000 [ 5631.896870] Lustre: server umount lustre-MDT0000 complete [ 5632.286056] LustreError: 167487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.053385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.055657] LustreError: 165039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.056932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5635.070144] Lustre: Skipped 2 previous similar messages [ 5635.098836] LustreError: 165039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5640.164363] LustreError: 165772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.202910] LustreError: 165772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5644.433348] LustreError: 165024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.42@tcp arrived at 1788157915 with bad export cookie 7506101560752180936 [ 5650.339398] Lustre: 164354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157905/real 1788157905] req@00000000c2431c97 x1875019362760192/t0(0) o400->MGC192.168.202.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788157921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5650.368666] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5650.380033] LustreError: 165040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.408119] LustreError: 165040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5663.782577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5670.882347] LustreError: 166197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.913477] LustreError: 166197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5676.318260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.400444] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5681.639362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5681.663349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5682.054849] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5684.822446] Lustre: lustre-MDT0000: Denying connection for new client abc015c8-0638-4d19-982d-488c50febc93 (at 192.168.202.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5695.456705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5695.474964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.490855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5696.999443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.000842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.601900] Lustre: server umount lustre-OST0000 complete [ 5703.600284] LustreError: 168780:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5703.617567] Lustre: 168130:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5703.626498] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5703.640722] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5703.652962] LustreError: 168130:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fd53cb2b x1875019362782848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5703.665223] LustreError: 168130:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5703.675991] LustreError: 168130:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5703.697628] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5704.295278] Lustre: server umount lustre-MDT0000 complete [ 5707.232445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.251793] LustreError: 166197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.287165] LustreError: 166197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5709.062213] LustreError: 165024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157979 with bad export cookie 7506101560752181265 [ 5709.072532] LustreError: 164353:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5709.076128] LustreError: 165024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5709.080133] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.719697] Lustre: server umount lustre-MDT0001 complete [ 5729.776975] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 5734.277642] Key type lgssc unregistered [ 5734.663363] LNet: 169535:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5734.668237] LNetError: 169535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5735.720543] LNet: Removed LNI 192.168.202.142@tcp [ 5736.710529] Key type .llcrypt unregistered [ 5736.712608] Key type ._llcrypt unregistered [ 5771.782103] Key type ._llcrypt registered [ 5771.784530] Key type .llcrypt registered [ 5771.866063] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 5773.306790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5773.323875] alg: No test for adler32 (adler32-zlib) [ 5774.424665] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5774.621818] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 5776.287327] Key type lgssc registered [ 5777.842956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.342199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.373927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.878055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.652584] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5797.346909] LustreError: 170952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.367604] LustreError: 170952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5802.167358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.637106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5808.410182] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5817.111590] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5824.385573] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5832.536241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.740221] LustreError: 172775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.982269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.327604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5840.628722] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5848.999131] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5857.767300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5857.771677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.806493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5857.823111] Lustre: Skipped 1 previous similar message [ 5857.841986] Lustre: Skipped 1 previous similar message [ 5862.901221] LustreError: 172777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.925873] LustreError: 172777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5863.150990] Lustre: server umount lustre-OST0000 complete [ 5868.000972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.017415] Lustre: Skipped 1 previous similar message [ 5868.031669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5868.045493] Lustre: Skipped 1 previous similar message [ 5873.122329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.542352] Lustre: server umount lustre-MDT0000 complete [ 5877.255604] LustreError: 170933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158147 with bad export cookie 852688555588867805 [ 5877.261349] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5877.279059] LustreError: 170933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5877.564688] Lustre: server umount lustre-MDT0001 complete [ 5891.786519] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 5894.430411] Key type lgssc unregistered [ 5894.716718] LNet: 174487:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5894.722700] LNetError: 174487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5894.735776] LNet: Removed LNI 192.168.202.142@tcp [ 5895.631407] Key type .llcrypt unregistered [ 5895.635310] Key type ._llcrypt unregistered [ 5932.174425] Key type ._llcrypt registered [ 5932.177601] Key type .llcrypt registered [ 5932.307585] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 5933.403478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5933.422412] alg: No test for adler32 (adler32-zlib) [ 5934.834717] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5935.177452] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 5936.807876] Key type lgssc registered [ 5938.496378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5947.440923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5947.484916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.303411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.436674] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5958.629600] LustreError: 176191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.654754] LustreError: 176191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5963.745123] LustreError: 176192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.708786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.194220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.963311] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5979.851372] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5987.119443] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5994.972316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5995.442689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.894900] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6009.081810] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6019.004659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6029.285549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.293122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6029.322356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6029.335646] Lustre: Skipped 1 previous similar message [ 6031.807713] Lustre: server umount lustre-OST0000 complete [ 6036.972143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6036.983578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.993682] Lustre: Skipped 1 previous similar message [ 6037.000183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.520408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.525952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.899376] Lustre: server umount lustre-MDT0000 complete [ 6044.640563] LustreError: 176186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6044.673493] LustreError: 176186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6047.548447] LustreError: 176170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158318 with bad export cookie 6270788637621258677 [ 6047.564582] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.589128] LustreError: 176170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6048.082479] Lustre: server umount lustre-MDT0001 complete [ 6060.729599] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6063.710658] Key type lgssc unregistered [ 6064.085079] LNet: 179748:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6064.097195] LNetError: 179748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6064.126127] LNet: Removed LNI 192.168.202.142@tcp [ 6065.169634] Key type .llcrypt unregistered [ 6065.170762] Key type ._llcrypt unregistered [ 6086.522824] Key type ._llcrypt registered [ 6086.524729] Key type .llcrypt registered [ 6086.632996] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6088.173873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6088.227496] alg: No test for adler32 (adler32-zlib) [ 6089.387633] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6089.697722] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6091.375184] Key type lgssc registered [ 6092.857517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6103.138617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6103.157987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.512592] Lustre: *** cfs_fail_loc=135, val=0*** [ 6104.575311] LustreError: 181252:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6104.578800] LustreError: 181252:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.142@tcp: cfg command failed: rc = -2 [ 6104.583639] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6104.583639] [ 6104.590059] LustreError: MGC192.168.202.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6104.597446] LustreError: 181213:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6104.604442] LustreError: 181213:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 6104.612046] LustreError: 181213:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 6104.623593] LustreError: 181213:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6104.705775] Lustre: server umount lustre-MDT0000 complete [ 6104.708943] LustreError: 181213:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6110.943918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.343494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.683652] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6116.834944] LustreError: 181625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.861669] LustreError: 181625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6121.954336] LustreError: 181624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.574956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.850592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6127.979910] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6135.502096] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6142.541856] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6146.537843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.546513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.560248] Lustre: Skipped 1 previous similar message [ 6146.577622] Lustre: Skipped 1 previous similar message [ 6148.642607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6150.861541] Lustre: server umount lustre-MDT0000 complete [ 6153.699033] LustreError: 182353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.730323] LustreError: 182353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6155.549914] LustreError: 181606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158426 with bad export cookie 15164437703930210074 [ 6155.556326] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.562743] LustreError: 181606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6156.105676] Lustre: server umount lustre-MDT0001 complete [ 6169.283860] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6172.816884] Key type lgssc unregistered [ 6173.164865] LNet: 183945:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6173.193060] LNetError: 183945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6174.247865] LNet: Removed LNI 192.168.202.142@tcp [ 6175.327351] Key type .llcrypt unregistered [ 6175.334119] Key type ._llcrypt unregistered [ 6197.226279] Key type ._llcrypt registered [ 6197.232908] Key type .llcrypt registered [ 6197.407551] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6197.770616] Key type .llcrypt unregistered [ 6197.772528] Key type ._llcrypt unregistered [ 6214.251188] Key type ._llcrypt registered [ 6214.253693] Key type .llcrypt registered [ 6214.369396] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6215.346503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6215.386325] alg: No test for adler32 (adler32-zlib) [ 6216.704509] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6217.066821] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6218.783253] Key type lgssc registered [ 6219.965242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6227.899953] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6227.947514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.616101] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6248.735340] LustreError: 186070:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 6275.359921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.489704] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6291.172681] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6299.622289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.718742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.858655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6305.761377] LustreError: 186095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.795332] LustreError: 186095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6306.787488] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6309.878982] LustreError: 187107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.909145] LustreError: 187107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6314.981917] LustreError: 187109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.008096] LustreError: 187109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6316.810114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.492114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6322.587966] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6332.395696] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6339.109835] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6342.375929] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6356.960626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6356.968133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.981969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6358.511094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6358.521501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.526874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.435368] Lustre: server umount lustre-OST0000 complete [ 6368.739268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.743167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6368.765109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.785823] Lustre: Skipped 2 previous similar messages [ 6371.250795] Lustre: server umount lustre-MDT0000 complete [ 6373.859285] LustreError: 187112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.900642] LustreError: 187112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6376.438866] LustreError: 186078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158647 with bad export cookie 9515396878813674234 [ 6376.444636] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.458444] LustreError: 186078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6377.031954] Lustre: server umount lustre-MDT0001 complete [ 6390.722281] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6393.290916] Key type lgssc unregistered [ 6393.528840] LNet: 190037:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6393.534910] LNetError: 190037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6393.556913] LNet: Removed LNI 192.168.202.142@tcp [ 6394.409775] Key type .llcrypt unregistered [ 6394.412169] Key type ._llcrypt unregistered [ 6423.145321] Key type ._llcrypt registered [ 6423.148027] Key type .llcrypt registered [ 6423.303957] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6425.396476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6425.551346] alg: No test for adler32 (adler32-zlib) [ 6426.881431] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6427.134088] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6428.799337] Key type lgssc registered [ 6430.859681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6439.461707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6439.477395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.000937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.887602] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6451.171054] LustreError: 191452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.183698] LustreError: 191452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6455.085299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.546326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6460.510895] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6469.500188] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6475.796928] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6484.382368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.949700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.007251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6493.246729] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6502.053953] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6508.461815] Lustre: Failing over lustre-MDT0000 [ 6509.014127] Lustre: server umount lustre-MDT0000 complete [ 6510.368255] LustreError: 191447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.568115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.591117] Lustre: Skipped 2 previous similar messages [ 6515.457465] LustreError: 193295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.488914] LustreError: 193295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6520.577113] LustreError: 193295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.591178] LustreError: 193295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6525.691148] LustreError: 191445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.715675] LustreError: 191445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6526.943367] Lustre: 190761:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788158781/real 1788158781] req@00000000c8942fc4 x1875020293892864/t0(0) o400->MGC192.168.202.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788158797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6526.971877] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.487299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.166077] LustreError: 192455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6536.182858] LustreError: 192455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6537.198501] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa83992d2dc8428c5 [ 6537.219522] Lustre: MGC192.168.202.142@tcp: Connection restored to 0@lo (at 0@lo) [ 6537.593909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6537.659570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6542.082748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6542.846612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6542.881691] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6542.962620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6543.296963] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6548.158433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6556.238705] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6563.303657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.309980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6563.327267] Lustre: Skipped 1 previous similar message [ 6564.364333] Lustre: server umount lustre-OST0000 complete [ 6573.536439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.541566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.545151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6573.545160] Lustre: Skipped 1 previous similar message [ 6573.563134] Lustre: Skipped 1 previous similar message [ 6574.790949] Lustre: server umount lustre-MDT0000 complete [ 6578.664126] LustreError: 195302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.691146] LustreError: 195302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6578.775087] LustreError: 191431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158849 with bad export cookie 12121881306384836805 [ 6578.776213] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6578.789479] LustreError: 191431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6579.328687] Lustre: server umount lustre-MDT0001 complete [ 6593.198824] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6596.189285] Key type lgssc unregistered [ 6596.631227] LNet: 196347:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6596.634487] LNetError: 196347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6596.671946] LNet: Removed LNI 192.168.202.142@tcp [ 6597.754070] Key type .llcrypt unregistered [ 6597.757468] Key type ._llcrypt unregistered [ 6623.818036] Key type ._llcrypt registered [ 6623.820598] Key type .llcrypt registered [ 6623.996121] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6625.155343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6625.191575] alg: No test for adler32 (adler32-zlib) [ 6626.152685] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6626.484650] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6628.271194] Key type lgssc registered [ 6629.088788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6637.571750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6637.616036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.585610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.361808] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6649.825136] LustreError: 197760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.855241] LustreError: 197760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6654.957512] LustreError: 197761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.073540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.497453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6664.544849] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6673.793970] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6680.253132] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6687.667254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.017300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6695.191780] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6703.590039] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6709.507553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6712.005452] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6714.410915] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6728.489624] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6735.330752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6735.344787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.354984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6735.363245] Lustre: Skipped 1 previous similar message [ 6740.452677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.970958] Lustre: server umount lustre-OST0000 complete [ 6745.581601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.583167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6745.604406] Lustre: Skipped 2 previous similar messages [ 6745.614911] Lustre: Skipped 1 previous similar message [ 6750.689206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6750.719654] Lustre: Skipped 2 previous similar messages [ 6755.809405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6755.813739] Lustre: Skipped 1 previous similar message [ 6759.393410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6759.716851] Lustre: server umount lustre-MDT0000 complete [ 6760.929418] LustreError: 197755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.960471] LustreError: 197755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6763.713341] LustreError: 197742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159034 with bad export cookie 13100601762610801052 [ 6763.716825] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6763.720349] LustreError: 197742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6764.217457] Lustre: server umount lustre-MDT0001 complete [ 6775.908140] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6779.341431] Key type lgssc unregistered [ 6779.590755] LNet: 201610:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6779.600606] LNetError: 201610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6779.635590] LNet: Removed LNI 192.168.202.142@tcp [ 6780.518170] Key type .llcrypt unregistered [ 6780.520572] Key type ._llcrypt unregistered [ 6803.304682] Key type ._llcrypt registered [ 6803.306228] Key type .llcrypt registered [ 6803.437566] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6805.037322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6805.049754] alg: No test for adler32 (adler32-zlib) [ 6806.091911] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6806.455027] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6808.183393] Key type lgssc registered [ 6809.638613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6817.566043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6817.608317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.084097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.688474] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6828.516795] LustreError: 202878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.535530] LustreError: 202878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6832.784689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.201630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.660951] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6847.169671] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6853.291354] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6861.279076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6861.907871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.993186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6868.602299] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6876.868089] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.879483] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6894.641565] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6904.566679] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6911.149263] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6920.674739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6920.684809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6920.699854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.743989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.751052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.758714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.778909] Lustre: Skipped 1 previous similar message [ 6925.527032] Lustre: server umount lustre-OST0000 complete [ 6930.402023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6930.420412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.433321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6933.984501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6933.984922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.884589] Lustre: server umount lustre-MDT0000 complete [ 6939.123477] LustreError: 202878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.147470] LustreError: 202878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6939.969992] LustreError: 206504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159210 with bad export cookie 13787779814806717590 [ 6939.980726] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6940.354755] Lustre: server umount lustre-MDT0001 complete [ 6953.484198] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6956.513464] Key type lgssc unregistered [ 6956.850861] LNet: 207658:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6956.855811] LNetError: 207658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6957.929885] LNet: Removed LNI 192.168.202.142@tcp [ 6958.919379] Key type .llcrypt unregistered [ 6958.925421] Key type ._llcrypt unregistered [ 6987.447599] Key type ._llcrypt registered [ 6987.449878] Key type .llcrypt registered [ 6987.577763] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6989.379328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6989.505443] alg: No test for adler32 (adler32-zlib) [ 6990.636872] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6991.101114] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6992.799371] Key type lgssc registered [ 6994.408453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7004.117194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7004.148096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.854192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.796206] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7015.407147] LustreError: 209088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.432965] LustreError: 209088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7020.517937] LustreError: 209087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.482768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.044297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7027.494970] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7036.025912] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7043.307905] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7052.509045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7052.822580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.417234] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7070.654066] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7076.445974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7091.680691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7091.685879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.700490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7091.707675] Lustre: Skipped 1 previous similar message [ 7094.984646] Lustre: server umount lustre-OST0000 complete [ 7101.922593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.939328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.955982] Lustre: Skipped 2 previous similar messages [ 7101.967882] Lustre: Skipped 1 previous similar message [ 7105.382527] Lustre: server umount lustre-MDT0000 complete [ 7107.042210] LustreError: 209082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7107.066049] LustreError: 209082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7109.381558] LustreError: 209821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159380 with bad export cookie 11950085363068266366 [ 7109.390300] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.397145] LustreError: 209821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7110.021156] Lustre: server umount lustre-MDT0001 complete [ 7121.871326] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7124.703184] Key type lgssc unregistered [ 7125.114849] LNet: 212773:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7125.128524] LNetError: 212773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7125.158116] LNet: Removed LNI 192.168.202.142@tcp [ 7126.033725] Key type .llcrypt unregistered [ 7126.036150] Key type ._llcrypt unregistered [ 7147.272668] Key type ._llcrypt registered [ 7147.278682] Key type .llcrypt registered [ 7147.518640] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7149.348545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7149.413236] alg: No test for adler32 (adler32-zlib) [ 7150.423232] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7150.740217] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7152.431210] Key type lgssc registered [ 7153.660636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7161.155653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7161.188138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.928714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7168.105901] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7173.091386] LustreError: 214046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.127637] LustreError: 214046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7177.891710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.210641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7178.401278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7183.086936] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7192.025210] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7198.238391] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7205.528464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7205.825682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.958063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7213.611259] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7223.034176] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.632636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.641771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7238.644030] Lustre: Skipped 1 previous similar message [ 7238.646134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7238.660191] Lustre: Skipped 1 previous similar message [ 7243.746560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7243.757291] Lustre: Skipped 1 previous similar message [ 7244.073339] Lustre: server umount lustre-OST0000 complete [ 7248.869292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.894629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7248.907573] Lustre: Skipped 1 previous similar message [ 7253.986410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7253.997585] Lustre: Skipped 1 previous similar message [ 7254.831103] Lustre: server umount lustre-MDT0000 complete [ 7258.713320] LustreError: 214027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159529 with bad export cookie 16534779514585250648 [ 7258.724318] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.220114] Lustre: server umount lustre-MDT0001 complete [ 7274.470545] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7278.563877] Key type lgssc unregistered [ 7279.030448] LNet: 217729:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7279.053916] LNetError: 217729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7279.092654] LNet: Removed LNI 192.168.202.142@tcp [ 7280.237053] Key type .llcrypt unregistered [ 7280.238629] Key type ._llcrypt unregistered [ 7302.125382] Key type ._llcrypt registered [ 7302.132574] Key type .llcrypt registered [ 7302.275372] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7303.644135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7303.755279] alg: No test for adler32 (adler32-zlib) [ 7305.213779] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7305.545871] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7307.280841] Key type lgssc registered [ 7308.627723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7317.702180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7317.731699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.491138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.673284] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7329.773331] LustreError: 218994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.807951] LustreError: 218994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7334.797761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.881911] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.301843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.308855] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7349.763947] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7356.095283] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7363.826280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7364.090060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.124818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7370.226172] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7379.009125] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7391.392756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.441904] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7391.742260] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7399.144883] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7408.175139] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7410.165473] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7422.311696] Lustre: Permanently deactivating lustre-OST0001 [ 7422.345379] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7428.199339] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7436.499818] Lustre: Permanently reactivating lustre-OST0001 [ 7436.531444] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7436.544439] Lustre: Skipped 2 previous similar messages [ 7446.126082] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.143039] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7446.154132] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7446.172887] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7446.248441] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7448.563920] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.579331] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7448.583981] Lustre: Skipped 2 previous similar messages [ 7448.602856] Lustre: Skipped 1 previous similar message [ 7449.666820] Lustre: server umount lustre-OST0001 complete [ 7453.670063] LustreError: 221546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.682349] LustreError: 221546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7456.227105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7456.238330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.260765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.259199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7457.268819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.788446] LustreError: 221546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.825673] LustreError: 221546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7460.513537] Lustre: server umount lustre-OST0000 complete [ 7468.514915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7468.524043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.546638] Lustre: Skipped 1 previous similar message [ 7468.557261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7468.564881] Lustre: Skipped 2 previous similar messages [ 7471.934889] Lustre: server umount lustre-MDT0000 complete [ 7474.145985] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7474.160287] LustreError: 218995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7475.788728] LustreError: 219731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159746 with bad export cookie 6204341038751045160 [ 7475.793365] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7475.802100] LustreError: 219731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7476.146110] Lustre: server umount lustre-MDT0001 complete [ 7488.229764] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7491.343815] Key type lgssc unregistered [ 7491.831411] LNet: 224056:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7491.841600] LNetError: 224056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7491.854936] LNet: Removed LNI 192.168.202.142@tcp [ 7492.774141] Key type .llcrypt unregistered [ 7492.776909] Key type ._llcrypt unregistered [ 7530.132256] Key type ._llcrypt registered [ 7530.140386] Key type .llcrypt registered [ 7530.251819] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7531.365961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7531.428976] alg: No test for adler32 (adler32-zlib) [ 7532.634849] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7532.867472] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7534.543197] Key type lgssc registered [ 7535.584278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7543.062361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7543.101352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7544.393742] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7544.433815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7544.635820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7548.789918] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7557.272370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.349465] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7557.378748] Lustre: 226261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7557.650977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7562.385349] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7570.142846] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7576.603811] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7584.284774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.493167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7584.687196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.882873] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7591.603629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7597.990821] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7600.896129] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7606.551197] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7614.147209] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7623.023516] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7632.689140] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7649.079478] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7649.082481] Lustre: Skipped 1 previous similar message [ 7683.524806] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7683.532285] Lustre: Skipped 4 previous similar messages [ 7752.672644] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7752.685470] Lustre: Skipped 8 previous similar messages [ 7769.060416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.070574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7769.076895] Lustre: Skipped 1 previous similar message [ 7769.840895] Lustre: server umount lustre-OST0000 complete [ 7774.179771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.183622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.200378] Lustre: Skipped 2 previous similar messages [ 7774.212862] Lustre: Skipped 1 previous similar message [ 7779.297252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7779.306301] Lustre: Skipped 1 previous similar message [ 7779.719107] Lustre: server umount lustre-MDT0000 complete [ 7783.777350] LustreError: 228938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160054 with bad export cookie 15793797432217536255 [ 7783.783823] LustreError: 228938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7783.783947] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7784.092151] Lustre: server umount lustre-MDT0001 complete [ 7796.692237] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7799.212215] Key type lgssc unregistered [ 7799.515480] LNet: 231039:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7799.523126] LNetError: 231039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7799.535960] LNet: Removed LNI 192.168.202.142@tcp [ 7800.261117] Key type .llcrypt unregistered [ 7800.264208] Key type ._llcrypt unregistered [ 7823.595660] Key type ._llcrypt registered [ 7823.596764] Key type .llcrypt registered [ 7823.680627] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7824.783781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7824.815444] alg: No test for adler32 (adler32-zlib) [ 7825.850568] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7826.166387] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7827.847238] Key type lgssc registered [ 7828.773854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7835.857780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7835.876329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.339034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.088305] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7847.917050] LustreError: 232453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7847.939078] LustreError: 232453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7848.457802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.817414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.149434] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7860.365457] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7865.970896] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7871.961267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.231342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7877.884423] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7883.503874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7884.895794] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7904.226584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.228634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.235407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7904.244082] Lustre: Skipped 1 previous similar message [ 7904.253888] Lustre: Skipped 1 previous similar message [ 7905.775341] Lustre: server umount lustre-OST0000 complete [ 7910.369141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7910.375875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.412448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7913.506728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7913.508689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.770367] Lustre: server umount lustre-MDT0000 complete [ 7918.564763] LustreError: 232448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.588043] LustreError: 232448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7920.299725] LustreError: 232433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160191 with bad export cookie 15275736081166387162 [ 7920.307758] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7920.322313] LustreError: 232433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7920.791469] Lustre: server umount lustre-MDT0001 complete [ 7932.704581] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7935.766972] Key type lgssc unregistered [ 7936.090714] LNet: 236151:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7936.099910] LNetError: 236151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7936.137418] LNet: Removed LNI 192.168.202.142@tcp [ 7937.378270] Key type .llcrypt unregistered [ 7937.385259] Key type ._llcrypt unregistered [ 7959.923488] Key type ._llcrypt registered [ 7959.925550] Key type .llcrypt registered [ 7960.046700] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7961.541203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7961.550667] alg: No test for adler32 (adler32-zlib) [ 7962.839624] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7963.183957] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7964.984599] Key type lgssc registered [ 7966.956621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8237.834428] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 8240.501386] Key type lgssc unregistered [ 8240.750942] LNet: 238172:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8240.759244] LNetError: 238172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8240.794765] LNet: Removed LNI 192.168.202.142@tcp [ 8241.493368] Key type .llcrypt unregistered [ 8241.494873] Key type ._llcrypt unregistered [ 8267.864994] Key type ._llcrypt registered [ 8267.867915] Key type .llcrypt registered [ 8268.036761] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 8283.273578] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8284.594907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8284.649765] alg: No test for adler32 (adler32-zlib) [ 8285.709325] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8285.928995] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 8287.623199] Key type lgssc registered [ 8288.825666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8318.739105] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8002 sec ========== 03:16:28 (1788160588) [ 8320.080644] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:16:29 (1788160589) === [ 8323.593964] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:16:32 (1788160592) === [ 8368.521213] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 8371.897855] Key type lgssc unregistered [ 8372.245671] LNet: 243428:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8372.257176] LNetError: 243428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8372.274593] LNet: Removed LNI 192.168.202.142@tcp [ 8373.180574] Key type .llcrypt unregistered [ 8373.182309] Key type ._llcrypt unregistered