[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512348237 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002297] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006916] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009125] LSM: Security Framework initializing [ 0.010053] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.021240] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026050] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030135] *** VALIDATE tmpfs *** [ 0.032323] *** VALIDATE proc *** [ 0.033278] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035308] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042559] debug: unmapping init [mem 0xffffffffa4259000-0xffffffffa4260fff] [ 0.044231] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045768] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053328] rcu: Hierarchical SRCU implementation. [ 0.055477] smp: Bringing up secondary CPUs ... [ 0.056507] x86: Booting SMP configuration: [ 0.057021] .... node #0, CPUs: #1 #2 #3 [ 0.060488] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145489] node 0 deferred pages initialised in 80ms [ 0.147306] devtmpfs: initialized [ 0.149266] x86/mm: Memory block size: 128MB [ 0.151927] gcov: version magic: 0x41383552 [ 0.155306] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160610] pinctrl core: initialized pinctrl subsystem [ 0.163197] [ 0.163754] ************************************************************* [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168011] ** ** [ 0.170014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172016] ** ** [ 0.174017] ** This means that this kernel is built to expose internal ** [ 0.177015] ** IOMMU data structures, which may compromise security on ** [ 0.179014] ** your system. ** [ 0.181016] ** ** [ 0.183018] ** If you see this message and you are not debugging the ** [ 0.185013] ** kernel, report this immediately to your vendor! ** [ 0.187014] ** ** [ 0.189013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191013] ************************************************************* [ 0.193694] NET: Registered protocol family 16 [ 0.195509] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204020] cpuidle: using governor menu [ 0.205824] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209598] PCI: Using configuration type 1 for base access [ 0.213183] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222207] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226102] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230593] cryptd: max_cpu_qlen set to 1000 [ 0.234172] ACPI: Added _OSI(Module Device) [ 0.236022] ACPI: Added _OSI(Processor Device) [ 0.238014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245826] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253637] ACPI: Interpreter enabled [ 0.255092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256019] ACPI: Using IOAPIC for interrupt routing [ 0.258155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.262481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274029] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.288489] acpiphp: Slot [2] registered [ 0.290168] acpiphp: Slot [5] registered [ 0.291188] acpiphp: Slot [6] registered [ 0.293145] acpiphp: Slot [7] registered [ 0.295119] acpiphp: Slot [8] registered [ 0.296140] acpiphp: Slot [9] registered [ 0.298144] acpiphp: Slot [10] registered [ 0.299132] acpiphp: Slot [3] registered [ 0.301138] acpiphp: Slot [4] registered [ 0.303133] acpiphp: Slot [11] registered [ 0.304106] acpiphp: Slot [12] registered [ 0.306206] acpiphp: Slot [13] registered [ 0.308128] acpiphp: Slot [14] registered [ 0.309118] acpiphp: Slot [15] registered [ 0.311156] acpiphp: Slot [16] registered [ 0.312161] acpiphp: Slot [17] registered [ 0.314127] acpiphp: Slot [18] registered [ 0.316122] acpiphp: Slot [19] registered [ 0.317119] acpiphp: Slot [20] registered [ 0.319159] acpiphp: Slot [21] registered [ 0.321121] acpiphp: Slot [22] registered [ 0.322162] acpiphp: Slot [23] registered [ 0.325200] acpiphp: Slot [24] registered [ 0.326123] acpiphp: Slot [25] registered [ 0.328117] acpiphp: Slot [26] registered [ 0.329122] acpiphp: Slot [27] registered [ 0.330141] acpiphp: Slot [28] registered [ 0.332134] acpiphp: Slot [29] registered [ 0.334128] acpiphp: Slot [30] registered [ 0.335112] acpiphp: Slot [31] registered [ 0.337071] PCI host bridge to bus 0000:00 [ 0.338024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.340027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347045] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354237] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.371023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377951] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.382035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388144] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.389584] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392077] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394678] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422072] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.473000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.490025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.520742] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537107] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.561019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.581069] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.587026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.610025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.623635] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.633020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.642020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.662025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682752] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.695019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.703018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.733022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.749970] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.752456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.754459] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.757571] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.759259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.765107] iommu: Default domain type: Passthrough [ 0.766429] SCSI subsystem initialized [ 0.767171] ACPI: bus type USB registered [ 0.769137] usbcore: registered new interface driver usbfs [ 0.771087] usbcore: registered new interface driver hub [ 0.773167] usbcore: registered new device driver usb [ 0.775225] pps_core: LinuxPPS API ver. 1 registered [ 0.777010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.780064] PTP clock support registered [ 0.782057] EDAC MC: Ver: 3.0.0 [ 0.783132] PCI: Using ACPI for IRQ routing [ 0.785032] NetLabel: Initializing [ 0.786011] NetLabel: domain hash size = 128 [ 0.788013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.790086] NetLabel: unlabeled traffic allowed by default [ 0.792309] vgaarb: loaded [ 0.794414] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.796028] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.802423] clocksource: Switched to clocksource kvm-clock [ 0.917380] VFS: Disk quotas dquot_6.6.0 [ 0.918812] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.921609] *** VALIDATE ramfs *** [ 0.922996] *** VALIDATE hugetlbfs *** [ 0.924676] pnp: PnP ACPI init [ 0.927363] pnp: PnP ACPI: found 6 devices [ 0.944388] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.947715] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.949519] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.951390] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.953896] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.956410] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.958605] NET: Registered protocol family 2 [ 0.961246] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.965981] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.969757] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.974399] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977130] TCP: Hash tables configured (established 65536 bind 65536) [ 0.979367] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.981422] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.984014] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.986357] NET: Registered protocol family 1 [ 0.988374] RPC: Registered named UNIX socket transport module. [ 0.990444] RPC: Registered udp transport module. [ 0.992190] RPC: Registered tcp transport module. [ 0.993640] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.995968] NET: Registered protocol family 44 [ 0.997665] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.999718] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.001672] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.003718] PCI: CLS 0 bytes, default 64 [ 1.005975] Unpacking initramfs... [ 2.439962] debug: unmapping init [mem 0xffff9a547cc54000-0xffff9a547ffbffff] [ 2.443953] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.446239] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.449174] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.920959] Initialise system trusted keyrings [ 2.922218] Key type blacklist registered [ 2.923568] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.931325] zbud: loaded [ 2.933717] *** VALIDATE nfs *** [ 2.934538] *** VALIDATE nfs4 *** [ 2.935747] pstore: using deflate compression [ 2.938490] Platform Keyring initialized [ 3.027395] NET: Registered protocol family 38 [ 3.028963] Key type asymmetric registered [ 3.030330] Asymmetric key parser 'x509' registered [ 3.031854] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.034331] io scheduler mq-deadline registered [ 3.035742] io scheduler kyber registered [ 3.037327] io scheduler bfq registered [ 3.038901] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.041554] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.044398] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.046605] ACPI: Power Button [PWRF] [ 3.051291] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.057363] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.072793] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.080098] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.093351] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.124365] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.154840] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.159704] Non-volatile memory driver v1.3 [ 3.161368] Linux agpgart interface v0.103 [ 3.192993] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.196439] vda: detected capacity change from 0 to 74694656 [ 3.232476] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.237119] vdb: detected capacity change from 0 to 1073741824 [ 3.258695] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.263302] vdc: detected capacity change from 0 to 2621440000 [ 3.280031] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.283034] vdd: detected capacity change from 0 to 2621440000 [ 3.298095] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.301221] vde: detected capacity change from 0 to 4294967296 [ 3.318064] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322237] vdf: detected capacity change from 0 to 4294967296 [ 3.331514] libphy: Fixed MDIO Bus: probed [ 3.339221] usbcore: registered new interface driver usbserial_generic [ 3.341545] usbserial: USB Serial support registered for generic [ 3.344236] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348796] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.350589] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.353237] mousedev: PS/2 mouse device common for all mice [ 3.356630] rtc_cmos 00:05: RTC can wake from S4 [ 3.359037] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.363109] rtc_cmos 00:05: registered as rtc0 [ 3.365260] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.366735] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.368528] intel_pstate: CPU model not supported [ 3.373542] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.374915] hid: raw HID events driver (C) Jiri Kosina [ 3.379808] usbcore: registered new interface driver usbhid [ 3.382743] usbhid: USB HID core driver [ 3.384941] drop_monitor: Initializing network drop monitor service [ 3.388404] Initializing XFRM netlink socket [ 3.391140] NET: Registered protocol family 10 [ 3.394719] Segment Routing with IPv6 [ 3.396792] NET: Registered protocol family 17 [ 3.400601] mpls_gso: MPLS GSO support [ 3.408078] RAS: Correctable Errors collector initialized. [ 3.410381] AVX version of gcm_enc/dec engaged. [ 3.412104] AES CTR mode by8 optimization enabled [ 3.502573] sched_clock: Marking stable (3502544577, 0)->(4419634897, -917090320) [ 3.506238] registered taskstats version 1 [ 3.508618] Loading compiled-in X.509 certificates [ 3.510411] zswap: loaded using pool lzo/zbud [ 3.536082] Key type big_key registered [ 3.551215] Key type encrypted registered [ 3.552837] ima: No TPM chip found, activating TPM-bypass! [ 3.554996] ima: Allocated hash algorithm: sha1 [ 3.556568] ima: No architecture policies found [ 3.558013] evm: Initialising EVM extended attributes: [ 3.559657] evm: security.selinux [ 3.560954] evm: security.ima [ 3.562044] evm: security.capability [ 3.563362] evm: HMAC attrs: 0x1 [ 3.566245] rtc_cmos 00:05: setting system clock to 2026-08-08 05:10:29 UTC (1786165829) [ 3.572469] debug: unmapping init [mem 0xffffffffa5203000-0xffffffffa53fffff] [ 3.575408] debug: unmapping init [mem 0xffffffffa3f82000-0xffffffffa4258fff] [ 3.584076] Write protecting the kernel read-only data: 28672k [ 3.587298] debug: unmapping init [mem 0xffffffffa2603000-0xffffffffa27fffff] [ 3.590294] debug: unmapping init [mem 0xffffffffa2f14000-0xffffffffa2ffffff] [ 3.628169] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.636662] systemd[1]: Detected virtualization kvm. [ 3.638500] systemd[1]: Detected architecture x86-64. [ 3.640491] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.667281] systemd[1]: No hostname configured. [ 3.669062] systemd[1]: Set hostname to . [ 3.671197] random: systemd: uninitialized urandom read (16 bytes read) [ 3.673564] systemd[1]: Initializing machine ID from random generator. [ 3.716263] random: ln: uninitialized urandom read (6 bytes read) [ 3.806424] random: systemd: uninitialized urandom read (16 bytes read) [ 3.809260] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.814249] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.818515] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.380409] device-mapper: uevent: version 1.0.3 [ 4.383140] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.070283] virtio_net virtio0 ens2: renamed from eth0 [ 5.070703] random: fast init done [ 5.130665] scsi host0: ata_piix [ 5.135812] scsi host1: ata_piix [ 5.137331] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.139724] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.509495] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.075061] random: crng init done [ 10.076405] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.451712] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.529526] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.801889] SELinux: Disabled at runtime. [ 11.861981] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.870438] systemd[1]: Detected virtualization kvm. [ 11.872303] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.357063] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.360521] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.365493] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.370438] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.374364] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.382137] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.390159] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.494829] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ 12.797043] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 13.071623] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.078146] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.214136] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.231774] EDAC sbridge: Ver: 1.1.2 [ 14.675781] Key type dns_resolver registered [ 14.974690] NFS: Registering the id_resolver key type [ 14.976476] Key type id_resolver registered [ 14.978220] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg145-server login: [ 40.873687] libcfs: loading out-of-tree module taints kernel. [ 40.897378] Key type ._llcrypt registered [ 40.898857] Key type .llcrypt registered [ 40.958774] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 60.294427] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 62.485836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.510260] alg: No test for adler32 (adler32-zlib) [ 64.220436] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 65.186026] hrtimer: interrupt took 2620757 ns [ 65.355168] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 67.143428] Key type lgssc registered [ 69.269784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.462122] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.552415] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 143.898633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 143.930922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.244277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 145.270962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 145.369776] Lustre: lustre-MDT0000: new disk, initializing [ 145.491671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.520195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.252269] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.740703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.869682] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.901625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.906324] Lustre: Skipped 1 previous similar message [ 163.969847] Lustre: lustre-MDT0001: new disk, initializing [ 164.033375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.055318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.077279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.670188] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.491178] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 182.450221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.633550] Lustre: lustre-OST0000: new disk, initializing [ 182.636329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 182.647485] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.735244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.987670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.033142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.140788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 189.121435] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.052886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.205019] Lustre: lustre-OST0001: new disk, initializing [ 203.208466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.213916] Lustre: 9514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.279702] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.992636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.026371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.107256] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 209.252967] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.149345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.714478] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.908425] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing check_logdir /tmp/testlogs/ [ 247.802574] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing yml_node [ 252.781842] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 255.571226] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 258.701904] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 260.335647] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 01:14:44 EDT 2026 [ 277.846068] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 279.496613] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.417244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.423090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.427224] Lustre: Skipped 3 previous similar messages [ 296.441109] Lustre: Skipped 3 previous similar messages [ 300.979427] Lustre: server umount lustre-MDT0000 complete [ 304.824445] LustreError: 7919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166130 with bad export cookie 6892026618027937616 [ 304.827440] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.831057] LustreError: 7919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 306.655740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.657072] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.657221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.671508] Lustre: Skipped 1 previous similar message [ 306.690312] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 309.664577] LustreError: 7460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.666516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.686512] LustreError: 7460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 309.696499] Lustre: Skipped 1 previous similar message [ 311.300626] Lustre: server umount lustre-MDT0001 complete [ 315.291233] Lustre: server umount lustre-OST0000 complete [ 319.531068] Lustre: server umount lustre-OST0001 complete [ 325.697252] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 334.378757] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 365.089963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.368551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 365.392824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 365.463318] Lustre: lustre-MDT0000: new disk, initializing [ 365.525286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.544418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 369.778424] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 380.274851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.399735] Lustre: 15064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 380.458290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 380.464854] Lustre: Skipped 1 previous similar message [ 380.575523] Lustre: lustre-MDT0001: new disk, initializing [ 380.683588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.711612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.727223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 385.696849] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 398.503761] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 405.432460] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 413.288085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 413.821582] Lustre: lustre-OST0000: new disk, initializing [ 413.824841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 413.830488] Lustre: 17036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 413.970956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.374645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.397056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.552198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.111397] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 432.486422] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.850180] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 438.090681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.235279] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.437752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.442612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.443859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.455993] LustreError: Skipped 1 previous similar message [ 446.461717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.464017] Lustre: Skipped 1 previous similar message [ 446.473778] Lustre: Skipped 2 previous similar messages [ 451.559911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.563382] Lustre: Skipped 1 previous similar message [ 451.888072] Lustre: server umount lustre-OST0000 complete [ 456.673606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.697223] Lustre: Skipped 1 previous similar message [ 461.793606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.798995] Lustre: Skipped 3 previous similar messages [ 462.867253] Lustre: server umount lustre-MDT0000 complete [ 466.916983] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.936575] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 467.431629] LustreError: 17035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166293 with bad export cookie 6892026618027939142 [ 467.454572] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.039642] Lustre: server umount lustre-MDT0001 complete [ 472.483843] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:18:16 (1786166296) [ 479.566851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.024678] LustreError: 19300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.170180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.349783] LustreError: 19301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.853823] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 494.566174] LustreError: 19300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.593474] LustreError: 19300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 496.001092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.604409] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 510.576979] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 516.610837] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 523.794224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.766494] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 538.928509] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 549.870947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.875182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.896248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.906080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 555.528873] Lustre: server umount lustre-OST0000 complete [ 562.144957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.164835] Lustre: Skipped 2 previous similar messages [ 566.499844] Lustre: server umount lustre-MDT0000 complete [ 567.264370] LustreError: 20028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.278826] LustreError: 20028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 569.907785] LustreError: 19282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166395 with bad export cookie 6892026618027940031 [ 569.921298] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.933581] LustreError: 19282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 570.681455] Lustre: server umount lustre-MDT0001 complete [ 581.690782] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 584.866391] Key type lgssc unregistered [ 585.133984] LNet: 22836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.148189] LNetError: 22836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.168167] LNet: Removed LNI 192.168.201.145@tcp [ 586.092245] Key type .llcrypt unregistered [ 586.096704] Key type ._llcrypt unregistered [ 605.766821] Key type ._llcrypt registered [ 605.772796] Key type .llcrypt registered [ 605.883937] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 606.902320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 606.996625] alg: No test for adler32 (adler32-zlib) [ 608.096506] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 608.408303] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 610.135334] Key type lgssc registered [ 611.595962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 619.323048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 619.349428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.814328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.791780] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 630.241120] LustreError: 24228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.274974] LustreError: 24228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 635.364681] LustreError: 24227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.894312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.147931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.125695] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 650.917357] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 656.529955] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 664.272698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.843844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.909049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 671.681676] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 679.328628] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 695.018350] Lustre: server umount lustre-OST0000 complete [ 701.922745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.933912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.952662] Lustre: Skipped 1 previous similar message [ 701.964427] Lustre: Skipped 1 previous similar message [ 707.040784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 707.050961] Lustre: Skipped 1 previous similar message [ 709.251702] Lustre: server umount lustre-MDT0000 complete [ 712.184611] LustreError: 24223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.186099] Lustre: 23539:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786166522/real 1786166522] req@0000000016ebaee8 x1872931254728320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786166538 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 712.202590] LustreError: 24223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 712.254959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.165894] LustreError: 24210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166539 with bad export cookie 723051814039592065 [ 713.187466] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.212800] Lustre: 23541:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786166522/real 1786166522] req@000000009ce74241 x1872931254728064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786166538 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [ 713.677892] Lustre: server umount lustre-MDT0001 complete [ 723.800502] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 726.562971] Key type lgssc unregistered [ 726.793986] LNet: 28208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.799979] LNetError: 28208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.808467] LNet: Removed LNI 192.168.201.145@tcp [ 727.579900] Key type .llcrypt unregistered [ 727.581765] Key type ._llcrypt unregistered [ 748.772920] Key type ._llcrypt registered [ 748.777952] Key type .llcrypt registered [ 748.893571] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 749.863979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 750.049178] alg: No test for adler32 (adler32-zlib) [ 751.084654] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 751.374392] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 753.032381] Key type lgssc registered [ 754.437245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 762.259521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 762.291416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.691606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 767.814976] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 773.107356] LustreError: 29603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.132148] LustreError: 29603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 777.123168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.724770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 782.841721] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 790.506980] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 796.569439] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 811.246759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 811.600969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.092861] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 827.462159] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 834.098355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 843.233119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 843.258949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.328688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 844.259053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.265675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 844.297597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 844.311706] Lustre: Skipped 1 previous similar message [ 846.454606] Lustre: server umount lustre-OST0000 complete [ 854.498042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.500679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 854.532364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.418878] Lustre: server umount lustre-MDT0000 complete [ 859.636131] LustreError: 29598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.651264] LustreError: 29598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 863.260757] LustreError: 29584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166689 with bad export cookie 6997462840769677153 [ 863.269526] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.755947] Lustre: server umount lustre-MDT0001 complete [ 873.167617] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 876.401580] Key type lgssc unregistered [ 876.909724] LNet: 33505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.932583] LNetError: 33505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.993023] LNet: Removed LNI 192.168.201.145@tcp [ 879.005559] Key type .llcrypt unregistered [ 879.009346] Key type ._llcrypt unregistered [ 904.725874] Key type ._llcrypt registered [ 904.727944] Key type .llcrypt registered [ 904.820033] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 905.668402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.813705] alg: No test for adler32 (adler32-zlib) [ 906.929609] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 907.179093] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 908.932225] Key type lgssc registered [ 910.214852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.930302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.943867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.473517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.408914] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 930.786195] LustreError: 34920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.797584] LustreError: 34920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 934.385811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.804102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 939.869281] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 947.614990] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.929807] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 961.123909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.587680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.697615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 968.943355] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 977.461820] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 990.176590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 990.185094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.188636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.193267] Lustre: Skipped 1 previous similar message [ 990.211350] Lustre: Skipped 1 previous similar message [ 993.570970] Lustre: server umount lustre-OST0000 complete [ 1000.417320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.428346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.439166] Lustre: Skipped 1 previous similar message [ 1000.457378] Lustre: Skipped 1 previous similar message [ 1005.519513] Lustre: server umount lustre-MDT0000 complete [ 1005.553761] LustreError: 34916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.577072] LustreError: 34916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1009.288673] LustreError: 34899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166835 with bad export cookie 12684166026366252977 [ 1009.294666] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.305341] LustreError: 34899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1009.736886] Lustre: server umount lustre-MDT0001 complete [ 1021.669720] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1026.008507] Key type lgssc unregistered [ 1026.461826] LNet: 38460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.476445] LNetError: 38460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.499238] LNet: Removed LNI 192.168.201.145@tcp [ 1027.998333] Key type .llcrypt unregistered [ 1028.001825] Key type ._llcrypt unregistered [ 1053.470066] Key type ._llcrypt registered [ 1053.471386] Key type .llcrypt registered [ 1053.539758] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1054.657664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.735836] alg: No test for adler32 (adler32-zlib) [ 1055.969146] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1056.286769] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1057.935431] Key type lgssc registered [ 1059.108654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.536711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.555663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.280115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.412145] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1079.782601] LustreError: 39874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.805441] LustreError: 39874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1083.292283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.668488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1088.876978] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1096.975804] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1102.868595] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1110.834973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.197309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.291838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1119.747795] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1129.836414] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1137.859486] Lustre: lustre-OST0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1139.174130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1139.175487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.188683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.205665] Lustre: Skipped 1 previous similar message [ 1139.229131] Lustre: Skipped 1 previous similar message [ 1142.963219] Lustre: lustre-OST0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1143.537365] Lustre: server umount lustre-OST0000 complete [ 1164.768641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.802285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.813066] Lustre: Skipped 1 previous similar message [ 1169.889404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.901292] Lustre: Skipped 1 previous similar message [ 1170.364946] Lustre: server umount lustre-MDT0000 complete [ 1174.261922] LustreError: 39855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167000 with bad export cookie 4046035996829604735 [ 1174.263112] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.275165] LustreError: 39855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1181.070962] Lustre: server umount lustre-MDT0001 complete [ 1193.571630] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1196.625627] Key type lgssc unregistered [ 1197.079129] LNet: 43587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.084944] LNetError: 43587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.103423] LNet: Removed LNI 192.168.201.145@tcp [ 1198.059746] Key type .llcrypt unregistered [ 1198.062634] Key type ._llcrypt unregistered [ 1225.827857] Key type ._llcrypt registered [ 1225.832993] Key type .llcrypt registered [ 1225.987901] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1227.255093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.356447] alg: No test for adler32 (adler32-zlib) [ 1228.420745] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1228.634461] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1230.303837] Key type lgssc registered [ 1231.520355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.379334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.416154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.853830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.315817] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1251.306810] LustreError: 45002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.337251] LustreError: 45002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1255.679994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.075037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.498936] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1270.246813] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1278.007370] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1285.015636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1285.336560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.405489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1291.973418] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1300.635373] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.378204] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1311.207275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.209869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.233402] Lustre: Skipped 2 previous similar messages [ 1311.254678] Lustre: Skipped 2 previous similar messages [ 1313.472124] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 1314.198257] Lustre: server umount lustre-MDT0000 complete [ 1316.327610] LustreError: 45002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.351550] LustreError: 45002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1318.382028] LustreError: 44983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167144 with bad export cookie 14303993453188745561 [ 1318.392534] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.401368] LustreError: 44983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1318.577774] LustreError: 44997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.614234] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 1321.445735] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.445847] LustreError: 44997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.488923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1323.498381] Lustre: Skipped 1 previous similar message [ 1325.029471] Lustre: server umount lustre-MDT0001 complete [ 1355.232692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1355.354518] Lustre: server umount lustre-OST0000 complete [ 1374.271451] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1377.276315] Key type lgssc unregistered [ 1377.582778] LNet: 48867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.591819] LNetError: 48867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.604866] LNet: Removed LNI 192.168.201.145@tcp [ 1378.607480] Key type .llcrypt unregistered [ 1378.609210] Key type ._llcrypt unregistered [ 1407.547383] Key type ._llcrypt registered [ 1407.549143] Key type .llcrypt registered [ 1407.662019] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1408.925416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.087897] alg: No test for adler32 (adler32-zlib) [ 1410.366218] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1410.648634] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1412.391196] Key type lgssc registered [ 1413.869149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1424.074441] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1424.093555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.714289] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1444.831535] LustreError: 50258:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 1471.522879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.834147] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1488.295467] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1657.044585] Lustre: server umount lustre-OST0000 complete [ 1678.834887] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1682.541461] Key type lgssc unregistered [ 1682.881314] LNet: 51937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.894655] LNetError: 51937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.919887] LNet: Removed LNI 192.168.201.145@tcp [ 1684.010267] Key type .llcrypt unregistered [ 1684.017842] Key type ._llcrypt unregistered [ 1708.143385] Key type ._llcrypt registered [ 1708.144898] Key type .llcrypt registered [ 1708.272695] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1709.278966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.419871] alg: No test for adler32 (adler32-zlib) [ 1710.633235] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1711.109465] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1712.823214] Key type lgssc registered [ 1714.486532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1726.560783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1726.575895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.929315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.694323] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1737.190763] LustreError: 53354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.210728] LustreError: 53354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1742.241425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.320160] LustreError: 53355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.651427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.575128] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1755.359410] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1762.431496] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1770.693765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.125635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.178271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1778.398951] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1787.390912] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1794.021053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.030564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.050360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1794.064811] LustreError: Skipped 1 previous similar message [ 1796.272969] Lustre: server umount lustre-OST0000 complete [ 1804.258238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.266760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1804.270072] Lustre: Skipped 1 previous similar message [ 1804.287968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.305085] Lustre: Skipped 2 previous similar messages [ 1806.433412] Lustre: server umount lustre-MDT0000 complete [ 1809.396924] LustreError: 54359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.428635] LustreError: 54359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1810.896811] LustreError: 54089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167636 with bad export cookie 14049451017163098372 [ 1810.901683] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.920471] LustreError: 54089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1811.440926] Lustre: server umount lustre-MDT0001 complete [ 1825.360234] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1828.231506] Key type lgssc unregistered [ 1828.540832] LNet: 56887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.547840] LNetError: 56887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.579279] LNet: Removed LNI 192.168.201.145@tcp [ 1829.779271] Key type .llcrypt unregistered [ 1829.782641] Key type ._llcrypt unregistered [ 1857.795077] Key type ._llcrypt registered [ 1857.797029] Key type .llcrypt registered [ 1857.845439] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1858.768644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1858.809645] alg: No test for adler32 (adler32-zlib) [ 1860.075334] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1860.407867] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1862.127194] Key type lgssc registered [ 1863.368953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.775747] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.795993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.312084] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1894.433510] LustreError: 58275:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 1921.247767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.041473] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1936.408576] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1942.611943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.708443] LustreError: 57611:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000aa3492e x1872932567517440/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1943.087237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.169706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1948.059552] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1948.136899] LustreError: 58302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.169925] LustreError: 58302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1953.250495] LustreError: 58301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.298549] LustreError: 59925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.327535] LustreError: 59925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1957.350353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.977102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.572567] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1972.898731] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.997907] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.459670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.477656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.495269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.505946] Lustre: Skipped 1 previous similar message [ 1988.454683] Lustre: server umount lustre-OST0000 complete [ 2010.888827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.924801] Lustre: Skipped 1 previous similar message [ 2010.938157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.940633] Lustre: Skipped 1 previous similar message [ 2016.226107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.241818] Lustre: Skipped 1 previous similar message [ 2017.307151] Lustre: server umount lustre-MDT0000 complete [ 2021.349048] LustreError: 59316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.365323] LustreError: 59316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2021.414735] LustreError: 58283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167847 with bad export cookie 11866724834944579128 [ 2021.432484] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.438247] LustreError: 58283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2027.959532] Lustre: server umount lustre-MDT0001 complete [ 2042.634992] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2046.229432] Key type lgssc unregistered [ 2046.661804] LNet: 62029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.668578] LNetError: 62029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.718694] LNet: Removed LNI 192.168.201.145@tcp [ 2048.782972] Key type .llcrypt unregistered [ 2048.792607] Key type ._llcrypt unregistered [ 2073.379387] Key type ._llcrypt registered [ 2073.383162] Key type .llcrypt registered [ 2073.512869] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2074.795035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.865295] alg: No test for adler32 (adler32-zlib) [ 2075.955418] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2076.261372] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2077.951176] Key type lgssc registered [ 2078.913758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2086.294568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2086.326610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.072032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.789460] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2097.634699] LustreError: 63443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.641373] LustreError: 63443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2102.323003] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.805958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2107.773811] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2116.153291] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2122.263863] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2129.348717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2129.684923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.790458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2135.335727] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2141.714964] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2157.536702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.537069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.544743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.556302] Lustre: Skipped 1 previous similar message [ 2159.378881] Lustre: server umount lustre-OST0000 complete [ 2164.194055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2164.198686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.214063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.219016] Lustre: Skipped 1 previous similar message [ 2167.776697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.777099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.033466] Lustre: server umount lustre-MDT0000 complete [ 2172.888468] LustreError: 63424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167998 with bad export cookie 1153761699841306457 [ 2172.899145] LustreError: 63424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2172.903666] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.215992] Lustre: server umount lustre-MDT0001 complete [ 2184.569073] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2187.121285] Key type lgssc unregistered [ 2187.413663] LNet: 66998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.426972] LNetError: 66998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.441884] LNet: Removed LNI 192.168.201.145@tcp [ 2188.359633] Key type .llcrypt unregistered [ 2188.361388] Key type ._llcrypt unregistered [ 2212.592513] Key type ._llcrypt registered [ 2212.594799] Key type .llcrypt registered [ 2212.727382] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2213.930114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2213.966339] alg: No test for adler32 (adler32-zlib) [ 2215.027400] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2215.200436] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2216.839183] Key type lgssc registered [ 2218.280730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2228.895782] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:47:33 (1786168053) [ 2237.088161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2237.114560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.773924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.301501] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2248.174848] LustreError: 68683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.193699] LustreError: 68683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2253.287797] LustreError: 68684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.514633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.916621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.390193] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2269.528798] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2275.430713] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2282.881987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2283.372623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.438933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2290.072971] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2298.077729] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2304.839596] Lustre: Failing over lustre-MDT0000 [ 2304.999288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.007281] Lustre: Skipped 1 previous similar message [ 2305.007920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.016750] Lustre: Skipped 2 previous similar messages [ 2305.419765] Lustre: server umount lustre-MDT0000 complete [ 2306.249482] LustreError: 68680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.117404] LustreError: 68683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.148393] LustreError: 68683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2315.234346] LustreError: 69412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.258224] LustreError: 69412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2315.326541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.430188] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.560706] Lustre: *** cfs_fail_loc=135, val=0*** [ 2315.681478] LustreError: 71771:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2315.688619] LustreError: 71771:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.145@tcp: cfg command failed: rc = -2 [ 2315.693499] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2315.693499] [ 2315.701548] LustreError: MGC192.168.201.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2315.707661] LustreError: 71760:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2315.711360] LustreError: 71760:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2315.716990] LustreError: 71760:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2315.722572] LustreError: 71760:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2315.744444] Lustre: server umount lustre-MDT0000 complete [ 2315.746470] LustreError: 71760:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2321.962832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.083875] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.456710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.507336] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2322.649502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2326.987204] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2327.535207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2327.584737] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2327.662763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2335.714104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.722514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.740919] Lustre: Skipped 1 previous similar message [ 2335.755805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.783265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.786116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.470682] Lustre: server umount lustre-OST0000 complete [ 2348.002933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.013106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.018281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.018291] Lustre: Skipped 1 previous similar message [ 2351.549145] Lustre: server umount lustre-MDT0000 complete [ 2355.291559] LustreError: 68663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168181 with bad export cookie 16060054751747635307 [ 2355.305235] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2355.315402] LustreError: 68663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2355.794996] Lustre: server umount lustre-MDT0001 complete [ 2367.323498] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2370.411663] Key type lgssc unregistered [ 2370.677864] LNet: 73555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2370.694592] LNetError: 73555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2370.715189] LNet: Removed LNI 192.168.201.145@tcp [ 2371.502624] Key type .llcrypt unregistered [ 2371.507593] Key type ._llcrypt unregistered [ 2393.022415] Key type ._llcrypt registered [ 2393.024369] Key type .llcrypt registered [ 2393.126199] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2393.948816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.071829] alg: No test for adler32 (adler32-zlib) [ 2395.132729] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2395.365752] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2397.055734] Key type lgssc registered [ 2398.260066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2406.973754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2407.005573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.447724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.939374] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2418.658661] LustreError: 74971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.666028] LustreError: 74971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2420.963105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.295352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.287761] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2437.245414] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2444.447690] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2453.870797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.061437] LustreError: 76791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.465295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.635326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2460.592085] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2468.876651] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2475.213434] Lustre: Failing over lustre-MDT0000 [ 2475.407942] Lustre: server umount lustre-MDT0000 complete [ 2476.751138] LustreError: 74967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.037285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.063406] Lustre: Skipped 1 previous similar message [ 2481.848026] LustreError: 76083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.877082] LustreError: 76083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2485.065506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.166831] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.293895] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2485.411749] LustreError: 78057:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2485.417628] LustreError: 78057:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.145@tcp: cfg command failed: rc = -9 [ 2485.424314] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2485.424314] [ 2485.436205] LustreError: MGC192.168.201.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2485.445164] LustreError: 78046:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2485.449195] LustreError: 78046:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2485.455300] LustreError: 78046:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2485.458734] LustreError: 78046:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2485.476804] Lustre: server umount lustre-MDT0000 complete [ 2485.479131] LustreError: 78046:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2486.956396] LustreError: 76083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.972281] LustreError: 76083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2491.578939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.691880] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.010096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.100446] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2492.117846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2497.004988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2497.039089] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2497.098649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2497.226043] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2507.243212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.245495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.274243] Lustre: Skipped 2 previous similar messages [ 2507.289754] Lustre: Skipped 1 previous similar message [ 2510.946645] Lustre: server umount lustre-OST0000 complete [ 2517.489532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.522286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.529363] Lustre: Skipped 1 previous similar message [ 2520.912810] Lustre: server umount lustre-MDT0000 complete [ 2522.592534] LustreError: 74967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.619381] LustreError: 74967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2524.917496] LustreError: 74950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168350 with bad export cookie 6846988278125383593 [ 2524.924354] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.927274] LustreError: 74950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2525.239463] Lustre: server umount lustre-MDT0001 complete [ 2538.381750] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2541.437209] Key type lgssc unregistered [ 2541.834832] LNet: 79839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2541.838910] LNetError: 79839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.887063] LNet: Removed LNI 192.168.201.145@tcp [ 2543.821537] Key type .llcrypt unregistered [ 2543.822965] Key type ._llcrypt unregistered [ 2556.888469] Key type ._llcrypt registered [ 2556.893557] Key type .llcrypt registered [ 2557.037727] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2557.359370] Key type .llcrypt unregistered [ 2557.360861] Key type ._llcrypt unregistered [ 2582.009542] Key type ._llcrypt registered [ 2582.011128] Key type .llcrypt registered [ 2582.107305] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2583.679276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.727542] alg: No test for adler32 (adler32-zlib) [ 2584.954573] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2585.185413] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2586.839233] Key type lgssc registered [ 2588.597869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2597.009944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2597.033263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.569637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.494495] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2608.114754] LustreError: 81745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.140160] LustreError: 81745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2613.059169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.221230] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2613.324610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2617.960221] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2626.264708] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2632.379618] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2639.864100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.210484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.320208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2647.080650] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2655.238741] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2669.030485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.032068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.033647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.038036] LustreError: Skipped 1 previous similar message [ 2669.076791] Lustre: Skipped 1 previous similar message [ 2672.802368] Lustre: server umount lustre-OST0000 complete [ 2679.268325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.274880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.288882] Lustre: Skipped 1 previous similar message [ 2679.301362] Lustre: Skipped 2 previous similar messages [ 2683.351234] Lustre: server umount lustre-MDT0000 complete [ 2684.384741] LustreError: 81746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.411099] LustreError: 81746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2687.120141] LustreError: 83589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168513 with bad export cookie 4721073491273368983 [ 2687.122084] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2687.127413] LustreError: 83589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2687.418418] Lustre: server umount lustre-MDT0001 complete [ 2700.375834] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2703.297676] Key type lgssc unregistered [ 2703.590658] LNet: 85283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2703.601133] LNetError: 85283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2703.615034] LNet: Removed LNI 192.168.201.145@tcp [ 2704.641854] Key type .llcrypt unregistered [ 2704.644964] Key type ._llcrypt unregistered [ 2728.970581] Key type ._llcrypt registered [ 2728.977060] Key type .llcrypt registered [ 2729.222862] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2731.563021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.683426] alg: No test for adler32 (adler32-zlib) [ 2732.710829] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2732.936914] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2734.551206] Key type lgssc registered [ 2735.387694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.824348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2744.834513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.485231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.792994] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2757.090908] LustreError: 86703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.115684] LustreError: 86703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2761.723705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.213299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2762.381132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2767.592508] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2775.903889] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2781.507097] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2789.059851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.392142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.481216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2796.824325] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2804.997373] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2818.019817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.021316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2818.030994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.996276] Lustre: server umount lustre-OST0000 complete [ 2828.262016] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.275305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.280548] Lustre: Skipped 2 previous similar messages [ 2828.294013] Lustre: Skipped 2 previous similar messages [ 2831.099711] Lustre: server umount lustre-MDT0000 complete [ 2833.377250] LustreError: 86697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.395552] LustreError: 86697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2834.773594] LustreError: 86684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168660 with bad export cookie 14373044980375016291 [ 2834.784987] LustreError: 86684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2834.787430] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.145529] Lustre: server umount lustre-MDT0001 complete [ 2846.206333] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2849.628278] Key type lgssc unregistered [ 2849.941395] LNet: 90241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2849.959253] LNetError: 90241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2850.983591] LNet: Removed LNI 192.168.201.145@tcp [ 2851.963325] Key type .llcrypt unregistered [ 2851.965537] Key type ._llcrypt unregistered [ 2872.554898] Key type ._llcrypt registered [ 2872.556986] Key type .llcrypt registered [ 2872.661226] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2873.490554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2873.627518] alg: No test for adler32 (adler32-zlib) [ 2874.709863] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2874.928722] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2876.559195] Key type lgssc registered [ 2877.562360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2884.593775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2884.614949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.134736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.023635] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2896.353297] LustreError: 91631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.360889] LustreError: 91631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2898.164683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.612925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.578580] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2913.955540] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2920.872136] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2929.076964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.427240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.549941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2936.240501] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2943.550477] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2959.328651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.337586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.342681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2959.357791] Lustre: Skipped 1 previous similar message [ 2964.371850] Lustre: server umount lustre-OST0000 complete [ 2969.571104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.577772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.584262] Lustre: Skipped 1 previous similar message [ 2969.605197] Lustre: Skipped 2 previous similar messages [ 2974.689400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.694405] Lustre: Skipped 1 previous similar message [ 2975.454186] Lustre: server umount lustre-MDT0000 complete [ 2979.825256] LustreError: 92358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.852101] LustreError: 92358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2979.915749] LustreError: 91612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168805 with bad export cookie 888179143675398721 [ 2979.918783] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.930382] LustreError: 91612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2980.307543] Lustre: server umount lustre-MDT0001 complete [ 2992.038412] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2994.886875] Key type lgssc unregistered [ 2995.266934] LNet: 95191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2995.280298] LNetError: 95191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2996.333651] LNet: Removed LNI 192.168.201.145@tcp [ 2997.245503] Key type .llcrypt unregistered [ 2997.250344] Key type ._llcrypt unregistered [ 3020.142298] Key type ._llcrypt registered [ 3020.144753] Key type .llcrypt registered [ 3020.244944] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3021.287757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3021.309363] alg: No test for adler32 (adler32-zlib) [ 3022.403930] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3022.609723] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3024.247843] Key type lgssc registered [ 3025.144897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3033.208276] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3033.235830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3048.863423] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3053.983717] LustreError: 96579:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 3080.671226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.214885] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3094.728841] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3099.910920] Lustre: server umount lustre-OST0000 complete [ 3109.784471] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:02:14 (1786168934) [ 3116.425818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.800152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.814572] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3122.150789] LustreError: 98232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.176774] LustreError: 98232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3126.246114] LustreError: 98231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.567558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.031111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3135.761747] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3143.485946] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3150.239719] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3157.250737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3157.824800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.931620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3164.159146] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3172.456587] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3190.763103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.765953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.780618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.327242] Lustre: server umount lustre-OST0000 complete [ 3200.999366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.019437] Lustre: Skipped 1 previous similar message [ 3201.029987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.034215] Lustre: Skipped 2 previous similar messages [ 3204.923857] Lustre: server umount lustre-MDT0000 complete [ 3206.124860] LustreError: 98232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.156889] LustreError: 98232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3209.009885] LustreError: 98212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169034 with bad export cookie 17509840822390707219 [ 3209.034035] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.035816] LustreError: 98212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3209.370096] Lustre: server umount lustre-MDT0001 complete [ 3221.618702] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3224.778419] Key type lgssc unregistered [ 3225.102739] LNet: 101925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.116671] LNetError: 101925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3225.134297] LNet: Removed LNI 192.168.201.145@tcp [ 3226.153952] Key type .llcrypt unregistered [ 3226.157432] Key type ._llcrypt unregistered [ 3265.896527] Key type ._llcrypt registered [ 3265.899874] Key type .llcrypt registered [ 3266.022554] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3266.981574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3267.013102] alg: No test for adler32 (adler32-zlib) [ 3268.050447] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3268.265294] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3269.903224] Key type lgssc registered [ 3271.106117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3277.328423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3277.349197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.914976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.107415] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3289.066977] LustreError: 104112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.100077] LustreError: 104112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3294.183023] LustreError: 104113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.157488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.836428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.806864] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3313.345560] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3318.713705] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3325.532436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.997969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.849336] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3340.749042] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3349.486870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3359.725274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.725763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.728296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.733028] Lustre: Skipped 1 previous similar message [ 3364.429271] Lustre: server umount lustre-OST0000 complete [ 3369.440083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3369.444932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.453087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3369.456412] Lustre: Skipped 1 previous similar message [ 3375.077191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.081015] Lustre: Skipped 1 previous similar message [ 3379.169539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3379.174111] Lustre: Skipped 2 previous similar messages [ 3382.239324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3382.440177] Lustre: server umount lustre-MDT0000 complete [ 3384.295438] LustreError: 104113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.313561] LustreError: 104113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3385.691496] LustreError: 104094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169211 with bad export cookie 16342494078682235151 [ 3385.695233] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3385.701299] LustreError: 104094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3385.976475] Lustre: server umount lustre-MDT0001 complete [ 3396.408142] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3399.144448] Key type lgssc unregistered [ 3399.430471] LNet: 107677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3399.434182] LNetError: 107677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3399.442277] LNet: Removed LNI 192.168.201.145@tcp [ 3400.204491] Key type .llcrypt unregistered [ 3400.206425] Key type ._llcrypt unregistered [ 3419.625383] Key type ._llcrypt registered [ 3419.630680] Key type .llcrypt registered [ 3419.758753] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3420.596518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3420.613074] alg: No test for adler32 (adler32-zlib) [ 3421.772914] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3422.030373] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3423.688051] Key type lgssc registered [ 3424.827329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3433.173965] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3433.227847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.799568] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3453.919944] LustreError: 108920:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 3480.543197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.709146] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3494.386418] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3501.880109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.055628] LustreError: MGC192.168.201.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3502.062723] LustreError: 109944:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3502.066487] LustreError: 109944:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3502.071648] LustreError: 109944:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3502.126448] Lustre: server umount lustre-MDT0000 complete [ 3502.128976] LustreError: 109944:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3519.555621] Lustre: server umount lustre-OST0000 complete [ 3528.807933] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3537.005115] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3573.795103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.005865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3574.026208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3574.145041] Lustre: lustre-MDT0000: new disk, initializing [ 3574.463216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.510639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3579.883511] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3591.195892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.385342] Lustre: 112736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3591.473774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3591.476746] Lustre: Skipped 1 previous similar message [ 3591.585685] Lustre: lustre-MDT0001: new disk, initializing [ 3591.816585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3591.880921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3591.894073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3596.277615] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3607.879657] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3613.309991] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3620.379147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.804390] Lustre: lustre-OST0000: new disk, initializing [ 3620.812703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3620.836584] Lustre: 114701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3620.998423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.583520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3627.597075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3627.690897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3627.919911] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3637.092808] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3641.411355] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3641.668350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3646.801618] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3647.106754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3653.601593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3653.602452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3653.615015] Lustre: Skipped 1 previous similar message [ 3653.628799] Lustre: Skipped 1 previous similar message [ 3655.544239] Lustre: server umount lustre-OST0000 complete [ 3663.329393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3663.341659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.360463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.480963] Lustre: server umount lustre-MDT0000 complete [ 3669.614054] LustreError: 112729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169495 with bad export cookie 5750144309310374216 [ 3669.619803] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3669.622727] LustreError: 112729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.097996] Lustre: server umount lustre-MDT0001 complete [ 3679.162540] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:11:43 (1786169503) [ 3683.023555] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3699.022128] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3708.990213] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3742.761271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.083990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3743.104094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3743.182542] Lustre: lustre-MDT0000: new disk, initializing [ 3743.243332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.253830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3748.103350] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3757.435833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.579305] Lustre: 119605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3757.628103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3757.631923] Lustre: Skipped 1 previous similar message [ 3757.700978] Lustre: lustre-MDT0001: new disk, initializing [ 3757.783448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3757.805321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3757.811575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3761.933552] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3773.273368] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3780.878245] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3788.197280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.459348] Lustre: lustre-OST0000: new disk, initializing [ 3788.462059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3788.467213] Lustre: 121575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3788.572856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.816103] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3797.065054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3797.074271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3797.177223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3804.418985] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3808.795512] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3809.094380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3813.221453] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3813.400636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3817.954387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3817.960117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.962132] LustreError: Skipped 1 previous similar message [ 3817.972943] Lustre: Skipped 1 previous similar message [ 3817.973782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3817.996149] Lustre: Skipped 2 previous similar messages [ 3821.215472] Lustre: server umount lustre-OST0000 complete [ 3828.195937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.198375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3828.227544] Lustre: Skipped 2 previous similar messages [ 3828.246909] Lustre: Skipped 1 previous similar message [ 3831.418455] Lustre: server umount lustre-MDT0000 complete [ 3833.333027] LustreError: 119617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.353493] LustreError: 119617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3835.085441] LustreError: 119599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169661 with bad export cookie 5750144309310375098 [ 3835.093685] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3835.095897] LustreError: 119599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3835.670549] Lustre: server umount lustre-MDT0001 complete [ 3843.629337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.972876] LustreError: 123792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.064432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.212218] LustreError: 123793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.230142] LustreError: 123793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3849.579512] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3853.351626] LustreError: 123792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.296464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.403302] LustreError: 123793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3862.800595] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3871.122283] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3876.912272] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3883.192167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3883.496599] LustreError: 125611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.932950] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3897.566654] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3905.089861] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3914.720704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3914.733416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.735560] LustreError: Skipped 1 previous similar message [ 3914.736196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.765339] Lustre: Skipped 1 previous similar message [ 3916.144254] Lustre: server umount lustre-OST0000 complete [ 3920.366932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3924.966506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.971268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.972316] Lustre: Skipped 2 previous similar messages [ 3924.997157] Lustre: Skipped 1 previous similar message [ 3926.702773] Lustre: server umount lustre-MDT0000 complete [ 3930.089366] LustreError: 123793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.112567] LustreError: 123793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3930.490200] LustreError: 123774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169756 with bad export cookie 5750144309310375980 [ 3930.505601] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.143315] Lustre: server umount lustre-MDT0001 complete [ 3942.911212] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3945.873230] Key type lgssc unregistered [ 3946.328335] LNet: 127424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3946.343811] LNetError: 127424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3946.370715] LNet: Removed LNI 192.168.201.145@tcp [ 3947.783254] Key type .llcrypt unregistered [ 3947.790643] Key type ._llcrypt unregistered [ 3966.851969] Key type ._llcrypt registered [ 3966.859719] Key type .llcrypt registered [ 3966.983929] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3982.074847] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3983.019857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3983.186744] alg: No test for adler32 (adler32-zlib) [ 3984.287328] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3984.580278] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3986.265498] Key type lgssc registered [ 3987.590741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4026.173387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4026.203769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.648419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4027.687456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4027.797743] Lustre: lustre-MDT0000: new disk, initializing [ 4027.933718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.966029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4033.543199] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4045.301286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.453951] Lustre: 130581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4045.538850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4045.543687] Lustre: Skipped 1 previous similar message [ 4045.663318] Lustre: lustre-MDT0001: new disk, initializing [ 4045.790824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.845884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4045.865888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4050.866663] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4061.482687] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4067.784495] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.762705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.209126] Lustre: lustre-OST0000: new disk, initializing [ 4076.215020] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4076.224781] Lustre: 132551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4076.336976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.934227] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4083.851305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4083.860039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4084.035343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4093.295357] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4098.191805] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4098.427411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4103.347458] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4103.532490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4109.799531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.801864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.809564] LustreError: Skipped 1 previous similar message [ 4109.835590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.065408] Lustre: server umount lustre-OST0000 complete [ 4117.477416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4117.485798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.498599] Lustre: Skipped 1 previous similar message [ 4117.502241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4117.508094] Lustre: Skipped 1 previous similar message [ 4120.033047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.033112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.614660] Lustre: server umount lustre-MDT0000 complete [ 4126.855487] LustreError: 130572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169952 with bad export cookie 6253029506690671158 [ 4126.862454] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.877148] LustreError: 130572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4127.140601] Lustre: server umount lustre-MDT0001 complete [ 4135.413843] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:19:20 (1786169960) [ 4142.796381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.438994] LustreError: 134963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.458473] LustreError: 134963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4143.531858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.657086] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4148.709432] LustreError: 134964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.824445] LustreError: 134963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.633028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.083985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.281932] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4172.506079] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4179.555629] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.624845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.645577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4182.651935] Lustre: Skipped 2 previous similar messages [ 4187.870924] Lustre: server umount lustre-MDT0000 complete [ 4188.133320] LustreError: 134964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.107068] LustreError: 134945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170018 with bad export cookie 6253029506690672040 [ 4192.119500] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4192.480344] Lustre: server umount lustre-MDT0001 complete [ 4202.471449] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:20:27 (1786170027) [ 4210.472587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.144531] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4231.264310] LustreError: 137386:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 4257.055341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.504484] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4273.137907] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4275.630909] Lustre: server umount lustre-OST0000 complete [ 4285.474956] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:21:49 (1786170109) [ 4293.012049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.499177] LustreError: 138834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.518241] LustreError: 138834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4293.581349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.340492] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4303.840347] LustreError: 138835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.855902] LustreError: 138835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4306.545740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.523235] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4319.828567] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.819461] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4333.213646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4333.589605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.596323] Lustre: Skipped 1 previous similar message [ 4340.195569] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4347.488214] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4362.211525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.217899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.236354] Lustre: Skipped 3 previous similar messages [ 4362.247811] Lustre: Skipped 1 previous similar message [ 4366.688942] Lustre: server umount lustre-MDT0000 complete [ 4367.334088] LustreError: 138834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.351078] LustreError: 138834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4370.152557] LustreError: 138815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170196 with bad export cookie 6253029506690672705 [ 4370.163062] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4370.163329] LustreError: 138815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4370.537377] Lustre: server umount lustre-MDT0001 complete [ 4384.771698] Lustre: server umount lustre-OST0000 complete [ 4396.810070] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:23:41 (1786170221) [ 4404.544134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.020870] LustreError: 142509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.095111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.684331] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4419.118143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.728217] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4434.835547] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4440.998883] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4450.433904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4458.492062] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4466.739522] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4471.898620] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4474.605104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4475.283554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4479.425166] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4479.672595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4485.092967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4485.117864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.156153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4485.179518] Lustre: Skipped 2 previous similar messages [ 4487.888334] Lustre: server umount lustre-OST0000 complete [ 4498.978233] Lustre: server umount lustre-MDT0000 complete [ 4500.459571] LustreError: 143238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.482973] LustreError: 143238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4502.063224] LustreError: 142490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170327 with bad export cookie 6253029506690673776 [ 4502.074164] LustreError: 142490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4502.074600] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4502.320720] Lustre: server umount lustre-MDT0001 complete [ 4511.081604] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:25:35 (1786170335) [ 4518.039805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4533.599645] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4538.719456] LustreError: 146691:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 4564.383279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.398875] Lustre: Skipped 2 previous similar messages [ 4572.339798] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4580.752263] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4588.089619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.774779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4594.047130] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4604.323748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.042027] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4618.090921] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4625.254700] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4631.190215] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4631.488402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4637.021568] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4637.261483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4640.756424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.759917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4640.780755] Lustre: Skipped 5 previous similar messages [ 4640.781379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4640.781383] LustreError: Skipped 1 previous similar message [ 4640.820440] Lustre: Skipped 4 previous similar messages [ 4645.377864] Lustre: server umount lustre-MDT0000 complete [ 4645.861858] LustreError: 147730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.875764] LustreError: 147730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4649.728839] LustreError: 146697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170475 with bad export cookie 6253029506690674798 [ 4649.742167] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4649.745506] LustreError: 146697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4650.047527] Lustre: server umount lustre-MDT0001 complete [ 4664.873299] Lustre: server umount lustre-OST0000 complete [ 4674.520794] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:28:18 (1786170498) [ 4681.501633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.186539] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4702.305974] LustreError: 150889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 4734.066803] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4741.377472] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4749.156981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.694190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4754.834752] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4764.701908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.859906] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4780.572475] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4786.923766] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4794.881403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4795.089852] Lustre: lustre-OST0001: new disk, initializing [ 4795.093673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4795.098814] Lustre: 153764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4802.627625] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4803.195471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4803.210652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4803.372974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4814.233511] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.006876] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4820.363294] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4826.840324] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4827.045478] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4831.715695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.724058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.731665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.733015] Lustre: Skipped 1 previous similar message [ 4831.765055] Lustre: Skipped 1 previous similar message [ 4835.764232] Lustre: server umount lustre-OST0000 complete [ 4844.513881] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.528312] LustreError: Skipped 1 previous similar message [ 4846.587416] Lustre: server umount lustre-OST0001 complete [ 4858.157241] Lustre: server umount lustre-MDT0000 complete [ 4863.785302] LustreError: 150897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170689 with bad export cookie 6253029506690675750 [ 4863.798321] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4863.804687] LustreError: 150897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4864.462885] Lustre: server umount lustre-MDT0001 complete [ 4894.965809] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:31:59 (1786170719) [ 4896.829452] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4898.872930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4900.849237] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:32:05 (1786170725) [ 4908.847683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.258940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4909.314464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4909.606898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.614281] Lustre: Skipped 6 previous similar messages [ 4914.677860] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4924.481319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.619792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4924.640382] Lustre: 157327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4930.086426] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4938.752914] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4945.937990] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4953.597625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4953.844760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4961.844769] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4963.497285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4969.983169] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4975.630515] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4975.847148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4981.337325] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4981.670062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4983.777538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4983.783821] LustreError: Skipped 1 previous similar message [ 4983.787087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.805838] Lustre: Skipped 4 previous similar messages [ 4983.811668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4983.814597] Lustre: Skipped 7 previous similar messages [ 4998.111387] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4998.652842] Lustre: server umount lustre-OST0000 complete [ 5024.986389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.556653] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5042.236636] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5048.939607] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5049.198192] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.206315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.212062] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5049.233440] Lustre: Skipped 1 previous similar message [ 5050.385676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5055.471309] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5055.692927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5062.074887] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff988607cbd000.ost_server_uuid 50 [ 5064.160430] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff988607cbd000.ost_server_uuid in FULL state after 0 sec [ 5064.352879] LustreError: 161136:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.45@tcp) returned error from blocking AST (req@00000000d1caeca9 x1872934794966400 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000009f63e424/0x56c73934d4557d80 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.45@tcp remote: 0x520c6297635e4a6 expref: 5 pid: 157333 timeout: 5164 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5064.411481] LustreError: lustre-MDT0000: A client on nid 192.168.201.45@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5064.423945] LustreError: 157323:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.45@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000009f63e424/0x56c73934d4557d80 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.45@tcp remote: 0x520c6297635e4a6 expref: 6 pid: 157333 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5069.805967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.812203] LustreError: Skipped 1 previous similar message [ 5075.770934] Lustre: server umount lustre-OST0000 complete [ 5094.367328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.045113] Lustre: server umount lustre-MDT0000 complete [ 5095.398772] LustreError: 157339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.416025] LustreError: 157339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5099.386668] LustreError: 158068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170925 with bad export cookie 6253029506690676926 [ 5099.387719] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5099.393826] LustreError: 158068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5105.903122] Lustre: server umount lustre-MDT0001 complete [ 5118.710924] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5121.558047] Key type lgssc unregistered [ 5121.877754] LNet: 163456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5121.893751] LNetError: 163456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5121.903772] LNet: Removed LNI 192.168.201.145@tcp [ 5122.793378] Key type .llcrypt unregistered [ 5122.795508] Key type ._llcrypt unregistered [ 5148.978467] Key type ._llcrypt registered [ 5148.981287] Key type .llcrypt registered [ 5149.200549] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5150.996527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5151.027055] alg: No test for adler32 (adler32-zlib) [ 5152.215565] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5152.515882] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5154.279369] Key type lgssc registered [ 5156.453336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5166.566176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5166.605734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.079801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.699578] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5177.316918] LustreError: 164870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.329262] LustreError: 164870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5182.436416] LustreError: 164869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.624912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.318534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5191.689689] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5200.958272] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5207.421450] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5215.634825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.197862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.282816] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5234.171129] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5237.696066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5244.508645] Lustre: Failing over lustre-MDT0000 [ 5244.842815] Lustre: server umount lustre-MDT0000 complete [ 5245.135354] LustreError: 164866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5246.432423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5246.446787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.972170] LustreError: 165598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.976564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.991179] LustreError: 165598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5253.091231] LustreError: 164864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5253.126374] LustreError: 164864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5257.473645] LustreError: 164850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.45@tcp arrived at 1786171083 with bad export cookie 6186812321312072677 [ 5263.327660] Lustre: 164181:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786171073/real 1786171073] req@0000000098d96466 x1872936019456896/t0(0) o400->MGC192.168.201.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786171089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5263.334218] LustreError: 165598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.350151] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.385238] LustreError: 165598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5265.707393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.956675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.045972] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5279.210110] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5279.236160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5279.796768] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5282.566952] Lustre: lustre-MDT0000: Denying connection for new client 053ad879-d06e-4325-bc88-edc2f4a3fd6c (at 192.168.201.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5287.606032] Lustre: lustre-MDT0000: Denying connection for new client 053ad879-d06e-4325-bc88-edc2f4a3fd6c (at 192.168.201.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 5294.579199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.585373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.595131] Lustre: Skipped 2 previous similar messages [ 5294.608715] Lustre: Skipped 1 previous similar message [ 5298.829413] Lustre: server umount lustre-OST0000 complete [ 5303.669523] LustreError: 168605:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5303.678347] Lustre: 167953:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5303.688286] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5303.717464] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5303.727868] LustreError: 167953:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004ea912ea x1872936019477120/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5303.736903] LustreError: 167953:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5303.749583] LustreError: 167953:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5303.761487] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5304.312975] Lustre: server umount lustre-MDT0000 complete [ 5304.807918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.834373] LustreError: 167135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.856285] LustreError: 167135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5309.015165] LustreError: 164849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171134 with bad export cookie 6186812321312073006 [ 5309.025625] LustreError: 164849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5309.046770] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5315.593607] Lustre: server umount lustre-MDT0001 complete [ 5327.886506] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5330.937728] Key type lgssc unregistered [ 5331.231148] LNet: 169364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5331.243788] LNetError: 169364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5331.263841] LNet: Removed LNI 192.168.201.145@tcp [ 5332.205597] Key type .llcrypt unregistered [ 5332.207995] Key type ._llcrypt unregistered [ 5359.968478] Key type ._llcrypt registered [ 5359.969962] Key type .llcrypt registered [ 5360.121445] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5361.682100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5361.754277] alg: No test for adler32 (adler32-zlib) [ 5362.875852] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5363.237846] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5364.959370] Key type lgssc registered [ 5366.550518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5375.741054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5375.758174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.349716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.219890] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5387.747400] LustreError: 170778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.755248] LustreError: 170778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5392.868024] LustreError: 170779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.654783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.281299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5400.980269] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5410.889580] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5418.362363] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.533137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.899702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.310866] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5443.737946] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5447.494193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5457.904147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.907938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5457.913044] Lustre: Skipped 1 previous similar message [ 5457.916137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5462.850644] Lustre: server umount lustre-OST0000 complete [ 5468.129703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.137270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5468.149140] Lustre: Skipped 1 previous similar message [ 5468.167890] Lustre: Skipped 2 previous similar messages [ 5473.262218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5473.273686] Lustre: Skipped 1 previous similar message [ 5473.773412] Lustre: server umount lustre-MDT0000 complete [ 5477.350590] LustreError: 170778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5477.381787] LustreError: 170778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5477.837584] LustreError: 170756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171303 with bad export cookie 14205735528760157228 [ 5477.867132] LustreError: 170756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5477.870549] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.253963] Lustre: server umount lustre-MDT0001 complete [ 5491.070448] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5494.269685] Key type lgssc unregistered [ 5494.627570] LNet: 174328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5494.631847] LNetError: 174328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5494.643279] LNet: Removed LNI 192.168.201.145@tcp [ 5495.791155] Key type .llcrypt unregistered [ 5495.797616] Key type ._llcrypt unregistered [ 5530.909424] Key type ._llcrypt registered [ 5530.911453] Key type .llcrypt registered [ 5531.013607] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5532.334262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5532.430049] alg: No test for adler32 (adler32-zlib) [ 5533.545459] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5533.881882] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5535.583201] Key type lgssc registered [ 5536.777342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5545.335772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5545.350944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.015907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.182728] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5556.202216] LustreError: 176030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.223951] LustreError: 176030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5561.329660] LustreError: 176031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.360419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.889135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5571.343832] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5579.221104] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5585.636585] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5594.306540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5594.884138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.873243] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5611.387248] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5616.381633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5624.812280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.819582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.831259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5626.862211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5626.869645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.869861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5629.214464] Lustre: server umount lustre-OST0000 complete [ 5636.581443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5636.594162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5636.622418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5636.625064] Lustre: Skipped 1 previous similar message [ 5642.219889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.230823] Lustre: Skipped 3 previous similar messages [ 5647.329846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5647.337367] Lustre: Skipped 1 previous similar message [ 5648.351550] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5648.590151] Lustre: server umount lustre-MDT0000 complete [ 5651.436860] LustreError: 177718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.516862] LustreError: 177718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5653.189684] LustreError: 176765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171479 with bad export cookie 18166709064380566136 [ 5653.192896] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.202530] LustreError: 176765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5653.516795] Lustre: server umount lustre-MDT0001 complete [ 5667.497505] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5670.212971] Key type lgssc unregistered [ 5670.562934] LNet: 179571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5670.587508] LNetError: 179571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5670.612835] LNet: Removed LNI 192.168.201.145@tcp [ 5671.535169] Key type .llcrypt unregistered [ 5671.538154] Key type ._llcrypt unregistered [ 5696.086408] Key type ._llcrypt registered [ 5696.090471] Key type .llcrypt registered [ 5696.289506] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5698.050791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5698.310777] alg: No test for adler32 (adler32-zlib) [ 5699.506695] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5699.935301] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5701.688057] Key type lgssc registered [ 5703.116905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5715.905831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5715.936689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.422061] Lustre: *** cfs_fail_loc=135, val=0*** [ 5717.585283] LustreError: 181095:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5717.590660] LustreError: 181095:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.145@tcp: cfg command failed: rc = -2 [ 5717.602128] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5717.602128] [ 5717.611208] LustreError: MGC192.168.201.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5717.626506] LustreError: 181055:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5717.631829] LustreError: 181055:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5717.637718] LustreError: 181055:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5717.641585] LustreError: 181055:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5717.745322] Lustre: server umount lustre-MDT0000 complete [ 5717.751951] LustreError: 181055:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5726.353375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.992095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5732.734987] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5736.422775] LustreError: 181466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.463848] LustreError: 181466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5741.551103] LustreError: 181467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.853517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.223410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5748.741945] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5758.427616] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5765.478292] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5768.675543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5768.690641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.720946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.193479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.220387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.233296] Lustre: Skipped 1 previous similar message [ 5770.279773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.293428] Lustre: Skipped 1 previous similar message [ 5774.254866] Lustre: server umount lustre-MDT0000 complete [ 5775.338816] LustreError: 181463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.365874] LustreError: 181463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5779.875314] LustreError: 181447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171605 with bad export cookie 12690527923144616363 [ 5779.889336] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.893298] LustreError: 181447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5780.501959] Lustre: server umount lustre-MDT0001 complete [ 5796.804688] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5801.134070] Key type lgssc unregistered [ 5801.565984] LNet: 183792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5801.579294] LNetError: 183792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5802.601686] LNet: Removed LNI 192.168.201.145@tcp [ 5804.076102] Key type .llcrypt unregistered [ 5804.082240] Key type ._llcrypt unregistered [ 5831.317958] Key type ._llcrypt registered [ 5831.322655] Key type .llcrypt registered [ 5831.553338] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5831.940890] Key type .llcrypt unregistered [ 5831.946531] Key type ._llcrypt unregistered [ 5852.802780] Key type ._llcrypt registered [ 5852.804227] Key type .llcrypt registered [ 5852.942821] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5853.916812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5853.971399] alg: No test for adler32 (adler32-zlib) [ 5855.000838] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5855.237363] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5856.911338] Key type lgssc registered [ 5858.532935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5868.067679] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5868.104340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5883.810148] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5888.928923] LustreError: 185933:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 5915.621994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.782650] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5934.418705] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5942.989662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.775546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.893755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5948.916711] LustreError: 185957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5948.961587] LustreError: 185957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5950.506498] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5951.017924] LustreError: 186992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.039654] LustreError: 186992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5956.071037] LustreError: 186971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.105456] LustreError: 186971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5961.212950] LustreError: 186992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.245616] LustreError: 186992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5961.910968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.613144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.298684] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5978.388536] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5986.818249] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5990.338472] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6003.684949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6003.688498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.690722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.696215] Lustre: Skipped 1 previous similar message [ 6008.805182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6008.816361] Lustre: Skipped 2 previous similar messages [ 6009.610555] Lustre: server umount lustre-OST0000 complete [ 6019.059264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.066792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6019.092461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.165225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.185969] Lustre: Skipped 2 previous similar messages [ 6029.280792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6030.303493] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6030.532663] Lustre: server umount lustre-MDT0000 complete [ 6033.442338] LustreError: 186976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.478574] LustreError: 186976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6035.539479] LustreError: 188595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171861 with bad export cookie 6232746504673037543 [ 6035.552310] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.986965] Lustre: server umount lustre-MDT0001 complete [ 6048.341554] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6052.470093] Key type lgssc unregistered [ 6052.846538] LNet: 189903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6052.854182] LNetError: 189903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6052.879229] LNet: Removed LNI 192.168.201.145@tcp [ 6054.117401] Key type .llcrypt unregistered [ 6054.128033] Key type ._llcrypt unregistered [ 6078.230883] Key type ._llcrypt registered [ 6078.233531] Key type .llcrypt registered [ 6078.430301] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6080.001138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6080.033295] alg: No test for adler32 (adler32-zlib) [ 6081.263868] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6081.524969] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6083.207202] Key type lgssc registered [ 6084.391449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6092.166899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6092.179818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.622310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.073667] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6104.046950] LustreError: 191316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.068150] LustreError: 191316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6107.250886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.799421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6113.016453] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6120.897603] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6126.311473] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6134.447226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6135.177781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.234178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6142.699528] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6151.715982] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6157.813114] Lustre: Failing over lustre-MDT0000 [ 6158.216728] Lustre: server umount lustre-MDT0000 complete [ 6158.825703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6158.833637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.844105] LustreError: 192312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.915041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.919631] LustreError: 191316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.941861] Lustre: Skipped 1 previous similar message [ 6162.988898] LustreError: 191316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6168.036186] LustreError: 191317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.065875] LustreError: 191317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6173.154131] LustreError: 192312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.177561] LustreError: 192312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6178.205209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.273450] Lustre: 190628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786171988/real 1786171988] req@000000007cb9b450 x1872936993582208/t0(0) o400->MGC192.168.201.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786172004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6178.316447] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6183.396328] LustreError: 192312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.456510] LustreError: 192312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6188.513382] LustreError: 190625:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000043a893c6 x1872936993586688/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6188.953929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.015595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6190.261528] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6194.188082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6194.211207] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6194.266270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6194.698239] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6199.566757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6205.511425] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6214.112588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6214.119000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.128151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6214.658184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6219.748519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6219.762030] Lustre: Skipped 1 previous similar message [ 6220.443408] Lustre: server umount lustre-OST0000 complete [ 6224.867194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6224.871480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.892769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6224.909056] Lustre: Skipped 2 previous similar messages [ 6224.936630] Lustre: Skipped 1 previous similar message [ 6229.984742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6229.994691] Lustre: Skipped 1 previous similar message [ 6230.664457] Lustre: server umount lustre-MDT0000 complete [ 6234.335534] LustreError: 194897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172060 with bad export cookie 2988701034792962781 [ 6234.340650] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6234.348137] LustreError: 194897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6234.668602] Lustre: server umount lustre-MDT0001 complete [ 6245.330336] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6247.863720] Key type lgssc unregistered [ 6248.150870] LNet: 196161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6248.153894] LNetError: 196161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6249.198716] LNet: Removed LNI 192.168.201.145@tcp [ 6250.234480] Key type .llcrypt unregistered [ 6250.238663] Key type ._llcrypt unregistered [ 6274.386113] Key type ._llcrypt registered [ 6274.388323] Key type .llcrypt registered [ 6274.566063] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6275.693272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.870044] alg: No test for adler32 (adler32-zlib) [ 6276.881848] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6277.094221] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6278.759188] Key type lgssc registered [ 6280.120432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6287.410876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6287.452829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.967600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.060172] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6298.090380] LustreError: 197551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.098275] LustreError: 197551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6303.209278] LustreError: 197552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.438386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.787597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6309.307390] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6316.774291] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.633809] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6329.177168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.714425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6333.817247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6336.741290] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6345.063827] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6349.243077] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6353.108226] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6370.054354] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6388.669715] Lustre: server umount lustre-OST0000 complete [ 6395.361327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.363805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6395.384609] Lustre: Skipped 1 previous similar message [ 6395.400857] Lustre: Skipped 1 previous similar message [ 6400.482976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6400.495650] Lustre: Skipped 1 previous similar message [ 6402.603633] Lustre: server umount lustre-MDT0000 complete [ 6405.600606] LustreError: 197552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.613798] LustreError: 199399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172232 with bad export cookie 8136705440160364822 [ 6406.622477] Lustre: 196867:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786172216/real 1786172216] req@0000000046b4e176 x1872937199123456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786172232 ref 1 fl Rpc:EXQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6406.632416] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6406.635297] LustreError: 199399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6406.680431] Lustre: 196867:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6417.110322] Lustre: server umount lustre-MDT0001 complete [ 6428.892692] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6432.496495] Key type lgssc unregistered [ 6432.790220] LNet: 201425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6432.799817] LNetError: 201425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6432.813806] LNet: Removed LNI 192.168.201.145@tcp [ 6433.528353] Key type .llcrypt unregistered [ 6433.530084] Key type ._llcrypt unregistered [ 6453.791752] Key type ._llcrypt registered [ 6453.793990] Key type .llcrypt registered [ 6453.895231] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6455.176369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6455.220775] alg: No test for adler32 (adler32-zlib) [ 6456.289808] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6456.622945] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6458.343349] Key type lgssc registered [ 6459.448651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6467.338868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6467.386317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.910514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6474.467506] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6478.307684] LustreError: 202691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.356848] LustreError: 202691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6483.428717] LustreError: 202692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.329262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.534836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6489.717327] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6499.530238] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6505.813267] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6513.916470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.248337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6519.907909] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6526.824707] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.808470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6548.294779] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6560.712220] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6569.202930] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6575.075080] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6584.812347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6584.814982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.821067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6584.841695] Lustre: Skipped 1 previous similar message [ 6589.634902] Lustre: server umount lustre-OST0000 complete [ 6595.044203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.054366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6595.056832] Lustre: Skipped 1 previous similar message [ 6600.081110] Lustre: server umount lustre-MDT0000 complete [ 6600.177601] LustreError: 202687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.195516] LustreError: 202687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6604.187692] LustreError: 205397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172430 with bad export cookie 1768102413435655791 [ 6604.189149] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.198094] LustreError: 205397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6604.525820] Lustre: server umount lustre-MDT0001 complete [ 6616.152328] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6619.085480] Key type lgssc unregistered [ 6619.489964] LNet: 207600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6619.511245] LNetError: 207600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6619.541588] LNet: Removed LNI 192.168.201.145@tcp [ 6620.392722] Key type .llcrypt unregistered [ 6620.395310] Key type ._llcrypt unregistered [ 6646.752562] Key type ._llcrypt registered [ 6646.755554] Key type .llcrypt registered [ 6646.855335] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6648.420825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6648.460389] alg: No test for adler32 (adler32-zlib) [ 6649.484249] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6649.748971] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6651.416919] Key type lgssc registered [ 6653.218449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6665.765240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6665.783121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.357202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6675.418028] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6677.480162] LustreError: 209023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.493117] LustreError: 209023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6682.593092] LustreError: 209024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.995537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.706053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6692.974460] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6702.743992] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6709.610869] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.344795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.778280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6725.889412] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6734.484469] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6737.207784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6757.859963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.868877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.890336] Lustre: Skipped 1 previous similar message [ 6759.785585] Lustre: server umount lustre-OST0000 complete [ 6764.514140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6764.520534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.554108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6764.558684] Lustre: Skipped 1 previous similar message [ 6768.097406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.103928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6770.670165] Lustre: server umount lustre-MDT0000 complete [ 6773.217728] LustreError: 209754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.265335] LustreError: 209754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6775.021606] LustreError: 209006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172600 with bad export cookie 4332003145271787810 [ 6775.025910] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6775.030366] LustreError: 209006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6775.651261] Lustre: server umount lustre-MDT0001 complete [ 6786.692094] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6789.968606] Key type lgssc unregistered [ 6790.353738] LNet: 212733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6790.359341] LNetError: 212733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6790.370114] LNet: Removed LNI 192.168.201.145@tcp [ 6791.593843] Key type .llcrypt unregistered [ 6791.596141] Key type ._llcrypt unregistered [ 6811.518506] Key type ._llcrypt registered [ 6811.523818] Key type .llcrypt registered [ 6811.719706] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6813.454949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6813.478127] alg: No test for adler32 (adler32-zlib) [ 6814.725143] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6815.031955] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6816.735171] Key type lgssc registered [ 6818.097181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6827.551356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6827.596674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.134272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.645211] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6839.288388] LustreError: 214001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.319823] LustreError: 214001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6844.384962] LustreError: 214002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.230633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.637303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6851.154081] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6859.677199] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6867.039744] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6874.937906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6875.771875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.554295] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6891.464270] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6899.328128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6920.164601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6920.190790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6920.203095] Lustre: Skipped 1 previous similar message [ 6925.281502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6925.302031] Lustre: Skipped 1 previous similar message [ 6925.896744] Lustre: server umount lustre-OST0000 complete [ 6930.409335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.417279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6930.419274] Lustre: Skipped 2 previous similar messages [ 6930.428758] Lustre: Skipped 1 previous similar message [ 6935.520932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6935.535511] Lustre: Skipped 1 previous similar message [ 6936.861824] Lustre: server umount lustre-MDT0000 complete [ 6940.545665] LustreError: 213982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172766 with bad export cookie 3446051367788163600 [ 6940.562124] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6940.849203] Lustre: server umount lustre-MDT0001 complete [ 6953.975419] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6957.571913] Key type lgssc unregistered [ 6957.894151] LNet: 217711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6957.904351] LNetError: 217711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6958.958706] LNet: Removed LNI 192.168.201.145@tcp [ 6960.063118] Key type .llcrypt unregistered [ 6960.065057] Key type ._llcrypt unregistered [ 6981.061744] Key type ._llcrypt registered [ 6981.064328] Key type .llcrypt registered [ 6981.156487] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6982.140967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6982.342944] alg: No test for adler32 (adler32-zlib) [ 6983.423673] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6983.603304] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6985.311180] Key type lgssc registered [ 6986.554770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6995.359772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6995.383241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.819420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.833985] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7006.182903] LustreError: 219000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.226231] LustreError: 219000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7010.188061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.674480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7014.528693] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7025.567697] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7032.837844] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7040.786855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.487864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.630472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7047.708912] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7055.699065] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7066.209984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.319984] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7066.514959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7073.447367] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7081.937589] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7084.197663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7096.666182] Lustre: Permanently deactivating lustre-OST0001 [ 7096.703718] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7106.621413] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7118.233535] Lustre: Permanently reactivating lustre-OST0001 [ 7118.286544] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7118.294220] Lustre: Skipped 2 previous similar messages [ 7128.646886] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.671894] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7128.681729] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7128.690944] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7128.764996] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7130.087283] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.090479] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7130.094461] Lustre: Skipped 2 previous similar messages [ 7131.362649] Lustre: server umount lustre-OST0001 complete [ 7135.204644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.215442] LustreError: 220823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.227769] LustreError: 220823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7135.234667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7135.237402] Lustre: Skipped 1 previous similar message [ 7140.324393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7140.327244] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.334480] Lustre: Skipped 1 previous similar message [ 7140.358916] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7141.438351] Lustre: server umount lustre-OST0000 complete [ 7149.025407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7149.037767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.066343] Lustre: Skipped 1 previous similar message [ 7149.074900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7152.269127] Lustre: server umount lustre-MDT0000 complete [ 7155.685769] LustreError: 218999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.711653] LustreError: 218999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7156.561829] LustreError: 218980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172982 with bad export cookie 16189091589420978456 [ 7156.574523] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.578036] LustreError: 218980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7157.127498] Lustre: server umount lustre-MDT0001 complete [ 7169.153840] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7172.078371] Key type lgssc unregistered [ 7172.448215] LNet: 224100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7172.452752] LNetError: 224100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7173.479092] LNet: Removed LNI 192.168.201.145@tcp [ 7174.434380] Key type .llcrypt unregistered [ 7174.437151] Key type ._llcrypt unregistered [ 7216.252370] Key type ._llcrypt registered [ 7216.254616] Key type .llcrypt registered [ 7216.371302] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7217.410496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7217.586219] alg: No test for adler32 (adler32-zlib) [ 7218.608685] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7218.931456] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7220.650155] Key type lgssc registered [ 7222.138328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7230.854702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7230.871708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7232.182263] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7232.201909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7232.358861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.380450] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7246.176120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.318258] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7246.363830] Lustre: 226306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7246.720970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7252.140513] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7260.390674] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7266.475976] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7274.035938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.234730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7274.420889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.377225] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7282.875970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7289.660780] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7293.925406] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7300.114265] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7306.050870] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7314.902284] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7324.464690] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7344.155311] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7344.164235] Lustre: Skipped 2 previous similar messages [ 7380.151930] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7380.160620] Lustre: Skipped 4 previous similar messages [ 7447.661363] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7447.669649] Lustre: Skipped 7 previous similar messages [ 7472.096448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7472.108898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7472.125853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7472.618109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7472.627501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7472.658309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.074647] Lustre: server umount lustre-OST0000 complete [ 7482.338206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7482.345780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.358856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.968199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.981360] Lustre: Skipped 2 previous similar messages [ 7493.101715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7493.141362] Lustre: Skipped 1 previous similar message [ 7494.113719] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7494.589807] Lustre: server umount lustre-MDT0000 complete [ 7501.792401] LustreError: 229039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173327 with bad export cookie 10680824991657772756 [ 7501.797662] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.802153] LustreError: 229039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7502.511547] Lustre: server umount lustre-MDT0001 complete [ 7519.960446] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7523.796164] Key type lgssc unregistered [ 7524.259298] LNet: 231123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7524.279339] LNetError: 231123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7524.310909] LNet: Removed LNI 192.168.201.145@tcp [ 7525.553270] Key type .llcrypt unregistered [ 7525.556193] Key type ._llcrypt unregistered [ 7553.042561] Key type ._llcrypt registered [ 7553.043861] Key type .llcrypt registered [ 7553.125510] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7554.151507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7554.185894] alg: No test for adler32 (adler32-zlib) [ 7555.333851] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7555.589868] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7557.207298] Key type lgssc registered [ 7558.655230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7567.535550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7567.556317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.179838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7574.103697] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7579.617880] LustreError: 232540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.648160] LustreError: 232540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7583.072897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.480055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7588.901081] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7600.308530] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7606.736551] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7614.901986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.364313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.622635] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7630.970680] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7639.916995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7665.648716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7665.658560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.680700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7665.698776] Lustre: Skipped 1 previous similar message [ 7666.728768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7666.742768] Lustre: Skipped 1 previous similar message [ 7671.461483] Lustre: server umount lustre-OST0000 complete [ 7677.921388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.940195] Lustre: Skipped 1 previous similar message [ 7677.948448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.954883] Lustre: Skipped 1 previous similar message [ 7682.656684] Lustre: server umount lustre-MDT0000 complete [ 7683.041296] LustreError: 233518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.057088] LustreError: 233518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7687.202720] LustreError: 232521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173513 with bad export cookie 9681271285904393332 [ 7687.204331] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7687.223243] LustreError: 232521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7687.661666] Lustre: server umount lustre-MDT0001 complete [ 7700.845555] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7703.824501] Key type lgssc unregistered [ 7704.131873] LNet: 236286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7704.138126] LNetError: 236286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7704.162530] LNet: Removed LNI 192.168.201.145@tcp [ 7705.018137] Key type .llcrypt unregistered [ 7705.024892] Key type ._llcrypt unregistered [ 7728.854294] Key type ._llcrypt registered [ 7728.855995] Key type .llcrypt registered [ 7729.045259] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7730.441027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7730.614865] alg: No test for adler32 (adler32-zlib) [ 7731.703746] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7732.035190] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7733.784693] Key type lgssc registered [ 7735.309345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8009.020443] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8012.193632] Key type lgssc unregistered [ 8012.531833] LNet: 238313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8012.538208] LNetError: 238313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8012.551939] LNet: Removed LNI 192.168.201.145@tcp [ 8013.357563] Key type .llcrypt unregistered [ 8013.363720] Key type ._llcrypt unregistered [ 8044.283386] Key type ._llcrypt registered [ 8044.284970] Key type .llcrypt registered [ 8044.416989] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 8062.814873] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8064.056475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8064.335266] alg: No test for adler32 (adler32-zlib) [ 8065.546583] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 8065.830709] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 8067.567134] Key type lgssc registered [ 8069.117925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8107.131302] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7845 sec ========== 03:25:31 (1786173931) [ 8109.057722] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:25:33 (1786173933) === [ 8113.607616] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:25:37 (1786173937) === [ 8166.610176] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8170.404694] Key type lgssc unregistered [ 8170.835803] LNet: 243557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8170.841510] LNetError: 243557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8170.865436] LNet: Removed LNI 192.168.201.145@tcp [ 8171.887875] Key type .llcrypt unregistered [ 8171.890462] Key type ._llcrypt unregistered