[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 559608366 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003466] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005022] kvm-guest: setup PV IPIs [ 0.008626] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011289] LSM: Security Framework initializing [ 0.014037] Yama: becoming mindful. [ 0.015187] SELinux: Initializing. [ 0.016097] *** VALIDATE selinux *** [ 0.027329] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033345] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.035159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037158] *** VALIDATE tmpfs *** [ 0.039017] *** VALIDATE proc *** [ 0.040267] *** VALIDATE cgroup *** [ 0.041022] *** VALIDATE cgroup2 *** [ 0.042319] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044053] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.046035] Spectre V2 : User space: Vulnerable [ 0.047013] Speculative Store Bypass: Vulnerable [ 0.050601] debug: unmapping init [mem 0xffffffff9a059000-0xffffffff9a060fff] [ 0.052853] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053830] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054031] ... version: 2 [ 0.055023] ... bit width: 48 [ 0.056019] ... generic registers: 4 [ 0.057018] ... value mask: 0000ffffffffffff [ 0.058035] ... max period: 00007fffffffffff [ 0.059021] ... fixed-purpose events: 3 [ 0.060020] ... event mask: 000000070000000f [ 0.061460] rcu: Hierarchical SRCU implementation. [ 0.063944] smp: Bringing up secondary CPUs ... [ 0.064852] x86: Booting SMP configuration: [ 0.065037] .... node #0, CPUs: #1 #2 #3 [ 0.069158] smp: Brought up 1 node, 4 CPUs [ 0.071019] smpboot: Max logical packages: 1 [ 0.072016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107306] node 0 deferred pages initialised in 33ms [ 0.112007] devtmpfs: initialized [ 0.113452] x86/mm: Memory block size: 128MB [ 0.116823] gcov: version magic: 0x41383552 [ 0.118361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120408] pinctrl core: initialized pinctrl subsystem [ 0.122241] [ 0.122891] ************************************************************* [ 0.125016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128019] ** ** [ 0.131021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.134021] ** ** [ 0.137020] ** This means that this kernel is built to expose internal ** [ 0.139016] ** IOMMU data structures, which may compromise security on ** [ 0.142020] ** your system. ** [ 0.145018] ** ** [ 0.148018] ** If you see this message and you are not debugging the ** [ 0.151022] ** kernel, report this immediately to your vendor! ** [ 0.153016] ** ** [ 0.156019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159019] ************************************************************* [ 0.163313] NET: Registered protocol family 16 [ 0.165806] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177044] cpuidle: using governor menu [ 0.179233] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182882] PCI: Using configuration type 1 for base access [ 0.185252] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194249] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199155] cryptd: max_cpu_qlen set to 1000 [ 0.202289] ACPI: Added _OSI(Module Device) [ 0.204021] ACPI: Added _OSI(Processor Device) [ 0.206093] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214113] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225112] ACPI: Interpreter enabled [ 0.227177] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229082] ACPI: Using IOAPIC for interrupt routing [ 0.231207] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236686] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250519] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261212] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268965] acpiphp: Slot [2] registered [ 0.271353] acpiphp: Slot [5] registered [ 0.273217] acpiphp: Slot [6] registered [ 0.275201] acpiphp: Slot [7] registered [ 0.277155] acpiphp: Slot [8] registered [ 0.279168] acpiphp: Slot [9] registered [ 0.281184] acpiphp: Slot [10] registered [ 0.283187] acpiphp: Slot [3] registered [ 0.285140] acpiphp: Slot [4] registered [ 0.287220] acpiphp: Slot [11] registered [ 0.289166] acpiphp: Slot [12] registered [ 0.291261] acpiphp: Slot [13] registered [ 0.293155] acpiphp: Slot [14] registered [ 0.295166] acpiphp: Slot [15] registered [ 0.297162] acpiphp: Slot [16] registered [ 0.299157] acpiphp: Slot [17] registered [ 0.301191] acpiphp: Slot [18] registered [ 0.303156] acpiphp: Slot [19] registered [ 0.305363] acpiphp: Slot [20] registered [ 0.307279] acpiphp: Slot [21] registered [ 0.309296] acpiphp: Slot [22] registered [ 0.312174] acpiphp: Slot [23] registered [ 0.314145] acpiphp: Slot [24] registered [ 0.316163] acpiphp: Slot [25] registered [ 0.318212] acpiphp: Slot [26] registered [ 0.320255] acpiphp: Slot [27] registered [ 0.323159] acpiphp: Slot [28] registered [ 0.325182] acpiphp: Slot [29] registered [ 0.327175] acpiphp: Slot [30] registered [ 0.329168] acpiphp: Slot [31] registered [ 0.334059] PCI host bridge to bus 0000:00 [ 0.336030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.339037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349101] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.356303] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360308] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364675] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.385070] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.389027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.392173] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396864] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399989] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.403062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.408158] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.415021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.431025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.438021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.445604] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478561] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491031] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502428] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.510027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.549918] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.560025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.568026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.586028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596982] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.603022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.609034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.612000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.612000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.612000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.619027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.634029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.718588] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.721562] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.723618] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726405] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728439] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734094] iommu: Default domain type: Passthrough [ 0.736607] SCSI subsystem initialized [ 0.737250] ACPI: bus type USB registered [ 0.739149] usbcore: registered new interface driver usbfs [ 0.741105] usbcore: registered new interface driver hub [ 0.743174] usbcore: registered new device driver usb [ 0.745406] pps_core: LinuxPPS API ver. 1 registered [ 0.747013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.750085] PTP clock support registered [ 0.752460] EDAC MC: Ver: 3.0.0 [ 0.753861] PCI: Using ACPI for IRQ routing [ 0.755495] NetLabel: Initializing [ 0.757014] NetLabel: domain hash size = 128 [ 0.759026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.762113] NetLabel: unlabeled traffic allowed by default [ 0.765347] vgaarb: loaded [ 0.767076] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.769059] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.776577] clocksource: Switched to clocksource kvm-clock [ 0.952286] VFS: Disk quotas dquot_6.6.0 [ 0.954436] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.957475] *** VALIDATE ramfs *** [ 0.959017] *** VALIDATE hugetlbfs *** [ 0.960967] pnp: PnP ACPI init [ 0.964060] pnp: PnP ACPI: found 6 devices [ 0.982240] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.986111] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.988750] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.991273] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.994191] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.997102] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.000566] NET: Registered protocol family 2 [ 1.003657] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.009607] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.013792] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.020249] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.024892] TCP: Hash tables configured (established 65536 bind 65536) [ 1.029139] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.033672] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037692] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.044846] NET: Registered protocol family 1 [ 1.048098] RPC: Registered named UNIX socket transport module. [ 1.050948] RPC: Registered udp transport module. [ 1.052885] RPC: Registered tcp transport module. [ 1.055111] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.058188] NET: Registered protocol family 44 [ 1.060170] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.062837] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.065365] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.068169] PCI: CLS 0 bytes, default 64 [ 1.070251] Unpacking initramfs... [ 2.933594] debug: unmapping init [mem 0xffff8908bcc54000-0xffff8908bffbffff] [ 2.938120] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.940637] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.943927] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.709878] Initialise system trusted keyrings [ 3.711870] Key type blacklist registered [ 3.713877] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.724040] zbud: loaded [ 3.727129] *** VALIDATE nfs *** [ 3.728416] *** VALIDATE nfs4 *** [ 3.730246] pstore: using deflate compression [ 3.782712] Platform Keyring initialized [ 4.217916] hrtimer: interrupt took 20811698 ns [ 4.255911] NET: Registered protocol family 38 [ 4.259029] Key type asymmetric registered [ 4.260357] Asymmetric key parser 'x509' registered [ 4.262176] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.265145] io scheduler mq-deadline registered [ 4.266716] io scheduler kyber registered [ 4.268240] io scheduler bfq registered [ 4.269763] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.272629] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.275445] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.279040] ACPI: Power Button [PWRF] [ 4.284237] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.291807] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.310796] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.321563] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.336847] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.363216] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.391610] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.396253] Non-volatile memory driver v1.3 [ 4.397778] Linux agpgart interface v0.103 [ 4.429696] virtio_blk virtio1: [vda] 134720 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.432196] vda: detected capacity change from 0 to 68976640 [ 4.468876] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.471738] vdb: detected capacity change from 0 to 1073741824 [ 4.500704] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.503972] vdc: detected capacity change from 0 to 2621440000 [ 4.524942] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.528604] vdd: detected capacity change from 0 to 2621440000 [ 4.544222] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.547949] vde: detected capacity change from 0 to 4294967296 [ 4.567663] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.571586] vdf: detected capacity change from 0 to 4294967296 [ 4.580147] libphy: Fixed MDIO Bus: probed [ 4.586406] usbcore: registered new interface driver usbserial_generic [ 4.589599] usbserial: USB Serial support registered for generic [ 4.594803] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.600374] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.602340] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.604559] mousedev: PS/2 mouse device common for all mice [ 4.608905] rtc_cmos 00:05: RTC can wake from S4 [ 4.613260] rtc_cmos 00:05: registered as rtc0 [ 4.615182] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.620585] intel_pstate: CPU model not supported [ 4.622912] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.626669] hid: raw HID events driver (C) Jiri Kosina [ 4.631373] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.636308] usbcore: registered new interface driver usbhid [ 4.636315] usbhid: USB HID core driver [ 4.636728] drop_monitor: Initializing network drop monitor service [ 4.636867] Initializing XFRM netlink socket [ 4.637248] NET: Registered protocol family 10 [ 4.646824] Segment Routing with IPv6 [ 4.651025] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.655658] NET: Registered protocol family 17 [ 4.656415] mpls_gso: MPLS GSO support [ 4.681775] RAS: Correctable Errors collector initialized. [ 4.686653] AVX version of gcm_enc/dec engaged. [ 4.688397] AES CTR mode by8 optimization enabled [ 4.844058] sched_clock: Marking stable (4844025489, 0)->(6112153154, -1268127665) [ 4.848887] registered taskstats version 1 [ 4.852608] Loading compiled-in X.509 certificates [ 4.855575] zswap: loaded using pool lzo/zbud [ 4.885162] Key type big_key registered [ 4.901694] Key type encrypted registered [ 4.903862] ima: No TPM chip found, activating TPM-bypass! [ 4.906595] ima: Allocated hash algorithm: sha1 [ 4.908911] ima: No architecture policies found [ 4.911207] evm: Initialising EVM extended attributes: [ 4.913711] evm: security.selinux [ 4.915280] evm: security.ima [ 4.916752] evm: security.capability [ 4.918539] evm: HMAC attrs: 0x1 [ 4.921872] rtc_cmos 00:05: setting system clock to 2026-03-02 06:47:01 UTC (1772434021) [ 4.932237] debug: unmapping init [mem 0xffffffff9b003000-0xffffffff9b1fffff] [ 4.936907] debug: unmapping init [mem 0xffffffff99d82000-0xffffffff9a058fff] [ 4.947067] Write protecting the kernel read-only data: 28672k [ 4.950809] debug: unmapping init [mem 0xffffffff98403000-0xffffffff985fffff] [ 4.953620] debug: unmapping init [mem 0xffffffff98d14000-0xffffffff98dfffff] [ 5.210884] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.225866] systemd[1]: Detected virtualization kvm. [ 5.229879] systemd[1]: Detected architecture x86-64. [ 5.233629] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.266597] systemd[1]: No hostname configured. [ 5.269436] systemd[1]: Set hostname to . [ 5.272168] random: systemd: uninitialized urandom read (16 bytes read) [ 5.276627] systemd[1]: Initializing machine ID from random generator. [ 5.462980] random: ln: uninitialized urandom read (6 bytes read) [ 5.579934] random: systemd: uninitialized urandom read (16 bytes read) [ 5.583308] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.589691] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.599903] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.508279] device-mapper: uevent: version 1.0.3 [ 6.511821] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK [ 7.367533] random: fast init done ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.655190] virtio_net virtio0 ens2: renamed from eth0 [ 9.150150] scsi host0: ata_piix [ 9.248479] scsi host1: ata_piix [ 9.250863] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.256318] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.511602] random: crng init done [ 13.513252] random: 7 urandom warning(s) missed due to ratelimiting [ 15.257489] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.210939] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.146772] printk: systemd: 25 output lines suppressed due to ratelimiting [ 20.484551] SELinux: Disabled at runtime. [ 20.647712] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.655075] systemd[1]: Detected virtualization kvm. [ 20.656707] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.550742] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.556841] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.566908] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.571307] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.576060] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.593355] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.611854] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ 23.166845] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.265506] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.324536] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.544036] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.710168] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.759419] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.200213] Key type dns_resolver registered [ 30.609753] NFS: Registering the id_resolver key type [ 30.612270] Key type id_resolver registered [ 30.614340] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg148-server login: [ 100.875279] libcfs: loading out-of-tree module taints kernel. [ 100.908484] Key type ._llcrypt registered [ 100.922229] Key type .llcrypt registered [ 101.095320] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 123.472466] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 125.711436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.735222] alg: No test for adler32 (adler32-zlib) [ 127.248593] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 128.186227] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 129.976185] Key type lgssc registered [ 131.863374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.014453] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.060638] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 222.538255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 222.611615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.156439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.222171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.335414] Lustre: lustre-MDT0000: new disk, initializing [ 224.480044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.511467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.270484] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.168519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 245.274883] Lustre: 6518:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 245.299108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 245.303833] Lustre: Skipped 1 previous similar message [ 245.397536] Lustre: lustre-MDT0001: new disk, initializing [ 245.476931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.530802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 245.555167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 250.630250] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.643572] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 265.429329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.913917] Lustre: lustre-OST0000: new disk, initializing [ 265.920596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 266.035587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 267.658343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 267.668979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 267.742220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 273.209620] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.900467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 291.142370] Lustre: lustre-OST0001: new disk, initializing [ 291.151952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 291.231717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.580375] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.123614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 301.147393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 301.268253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.256833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 320.416322] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 334.179234] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing check_logdir /tmp/testlogs/ [ 340.697126] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing yml_node [ 346.000499] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 349.422660] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 352.596738] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 354.973311] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 01:52:49 EST 2026 [ 378.091863] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 379.773346] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 398.828496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.829750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 398.845673] Lustre: Skipped 2 previous similar messages [ 398.862654] Lustre: Skipped 3 previous similar messages [ 402.879260] Lustre: server umount lustre-MDT0000 complete [ 406.753533] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772434423 with bad export cookie 4787420071832749600 [ 406.758452] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.764635] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 409.061470] LustreError: 8431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.061664] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.073177] LustreError: 8431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 409.094249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 409.094865] Lustre: Skipped 2 previous similar messages [ 409.112653] Lustre: Skipped 1 previous similar message [ 411.104869] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.104887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 411.143818] Lustre: Skipped 1 previous similar message [ 416.225310] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.226180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 416.247350] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 416.272170] Lustre: Skipped 1 previous similar message [ 421.346105] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.947823] Lustre: server umount lustre-MDT0001 complete [ 427.400318] Lustre: server umount lustre-OST0000 complete [ 432.118563] Lustre: server umount lustre-OST0001 complete [ 441.237906] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 454.871736] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 500.594384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.947926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 501.017377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 501.142535] Lustre: lustre-MDT0000: new disk, initializing [ 501.252237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.277150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 506.218079] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 518.512692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.686241] Lustre: 14998:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 518.778820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 518.783951] Lustre: Skipped 1 previous similar message [ 518.885387] Lustre: lustre-MDT0001: new disk, initializing [ 518.975037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 519.018063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 519.034370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 523.701489] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 531.876179] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 536.432600] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 545.655497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.200690] Lustre: lustre-OST0000: new disk, initializing [ 546.206229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 546.356042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.825460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 547.836333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 548.016271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 554.056230] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 563.902980] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 569.375965] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 569.602226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 575.293892] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 575.517760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 578.529759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 578.540978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.551583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.563268] Lustre: Skipped 1 previous similar message [ 578.592742] Lustre: Skipped 3 previous similar messages [ 588.771916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.784334] Lustre: Skipped 3 previous similar messages [ 592.352159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 592.527945] Lustre: server umount lustre-OST0000 complete [ 599.011407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.043359] Lustre: Skipped 1 previous similar message [ 609.253996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.260276] Lustre: Skipped 5 previous similar messages [ 611.299906] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 611.562756] Lustre: server umount lustre-MDT0000 complete [ 614.369721] LustreError: 15858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.391214] LustreError: 15858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 616.085181] LustreError: 14990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772434632 with bad export cookie 4787420071832751133 [ 616.096137] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.097346] LustreError: 14990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 616.490592] Lustre: server umount lustre-MDT0001 complete [ 620.773482] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:57:15 (1772434635) [ 629.137276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.831791] LustreError: 18865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.927255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.332981] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 639.466956] LustreError: 18866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.488896] LustreError: 18866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 645.989437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.641547] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 659.420930] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 664.808277] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 673.391643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.260485] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 689.661208] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 701.923848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 701.937964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.945379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 701.968815] Lustre: Skipped 1 previous similar message [ 707.600901] Lustre: server umount lustre-OST0000 complete [ 712.176271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.201577] Lustre: Skipped 1 previous similar message [ 718.586589] Lustre: server umount lustre-MDT0000 complete [ 722.403307] LustreError: 20031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.424574] LustreError: 20031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 722.877696] LustreError: 18845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772434739 with bad export cookie 4787420071832752015 [ 722.896772] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.253552] Lustre: server umount lustre-MDT0001 complete [ 736.167308] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 739.388554] Key type lgssc unregistered [ 739.682877] LNet: 22046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.694464] LNetError: 22046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.718525] LNet: Removed LNI 192.168.201.148@tcp [ 741.995196] Key type .llcrypt unregistered [ 742.001498] Key type ._llcrypt unregistered [ 772.447706] Key type ._llcrypt registered [ 772.452920] Key type .llcrypt registered [ 772.653117] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 774.518846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.577401] alg: No test for adler32 (adler32-zlib) [ 775.800026] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 776.111898] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 777.840175] Key type lgssc registered [ 779.168823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 788.643806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 788.735976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.628824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.955392] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 800.740316] LustreError: 23445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.763686] LustreError: 23445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 805.863615] LustreError: 23446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.367430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.822466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 815.143666] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 822.296898] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 827.334668] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 836.262948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.800600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.313132] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 851.941756] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 861.384688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 871.906605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.907182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.914986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.929813] Lustre: Skipped 1 previous similar message [ 876.924760] Lustre: server umount lustre-OST0000 complete [ 882.152891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.168286] Lustre: Skipped 1 previous similar message [ 882.171454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 882.179358] Lustre: Skipped 2 previous similar messages [ 887.267058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.274363] Lustre: Skipped 1 previous similar message [ 888.309186] Lustre: server umount lustre-MDT0000 complete [ 892.393536] LustreError: 23440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.420130] LustreError: 23440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 892.807910] LustreError: 24165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772434909 with bad export cookie 1928536561276616245 [ 892.856292] LustreError: 24165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 892.875379] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.330147] Lustre: server umount lustre-MDT0001 complete [ 907.634737] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 910.696695] Key type lgssc unregistered [ 911.005180] LNet: 27083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.018233] LNetError: 27083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.035863] LNet: Removed LNI 192.168.201.148@tcp [ 912.158092] Key type .llcrypt unregistered [ 912.161434] Key type ._llcrypt unregistered [ 942.032449] Key type ._llcrypt registered [ 942.036332] Key type .llcrypt registered [ 942.200576] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 944.423883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.511256] alg: No test for adler32 (adler32-zlib) [ 945.807150] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 946.304503] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 948.090121] Key type lgssc registered [ 949.754880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.333128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.350866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.945208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.977933] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 972.264063] LustreError: 28482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.293618] LustreError: 28482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 977.389853] LustreError: 28481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.578432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.452888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.816920] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 992.223345] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 997.107210] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1013.434584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.017173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.294431] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1027.607427] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1032.447871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1039.842620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.857452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.883605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.918404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.918560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1042.923956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.560941] Lustre: server umount lustre-OST0000 complete [ 1051.112701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1051.127676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.139537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.151534] Lustre: Skipped 1 previous similar message [ 1053.165503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.166270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.214621] Lustre: Skipped 1 previous similar message [ 1056.459550] Lustre: server umount lustre-MDT0000 complete [ 1058.274056] LustreError: 29408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.303737] LustreError: 29408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1061.420049] LustreError: 28462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772435077 with bad export cookie 13638137331265046725 [ 1061.430589] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.441400] LustreError: 28462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1061.909824] Lustre: server umount lustre-MDT0001 complete [ 1075.013151] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1078.430086] Key type lgssc unregistered [ 1078.787935] LNet: 32009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.800381] LNetError: 32009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.826496] LNet: Removed LNI 192.168.201.148@tcp [ 1079.946689] Key type .llcrypt unregistered [ 1079.949938] Key type ._llcrypt unregistered [ 1108.652662] Key type ._llcrypt registered [ 1108.661084] Key type .llcrypt registered [ 1108.822553] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1110.502034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.572798] alg: No test for adler32 (adler32-zlib) [ 1111.672984] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1112.044316] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1113.785866] Key type lgssc registered [ 1115.470801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1124.607398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1124.639278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.497685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.894948] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1136.612885] LustreError: 33408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.646212] LustreError: 33408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1141.730507] LustreError: 33407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.814861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.488136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1149.692681] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1156.169082] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1161.368626] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1169.691278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.345647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.414426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1178.127929] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1185.231316] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1203.173913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.174079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.182079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.214050] Lustre: Skipped 1 previous similar message [ 1205.409920] Lustre: server umount lustre-OST0000 complete [ 1209.826644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.830697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.854589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.872609] Lustre: Skipped 1 previous similar message [ 1213.436941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.437604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.339769] Lustre: server umount lustre-MDT0000 complete [ 1218.531140] LustreError: 33407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.558799] LustreError: 33407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1219.557814] LustreError: 33389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772435236 with bad export cookie 7558579603208542568 [ 1219.559877] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1219.598031] LustreError: 33389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1220.658949] Lustre: server umount lustre-MDT0001 complete [ 1234.252639] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1236.711115] Key type lgssc unregistered [ 1237.159763] LNet: 36612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.164186] LNetError: 36612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.203719] LNet: Removed LNI 192.168.201.148@tcp [ 1238.148660] Key type .llcrypt unregistered [ 1238.153085] Key type ._llcrypt unregistered [ 1272.879981] Key type ._llcrypt registered [ 1272.882692] Key type .llcrypt registered [ 1273.031449] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1274.976552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.044138] alg: No test for adler32 (adler32-zlib) [ 1276.541902] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1276.897609] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1278.720284] Key type lgssc registered [ 1280.523391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1291.129428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1291.160371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.031669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.029675] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1302.505185] LustreError: 38011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.530274] LustreError: 38011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1307.245091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.787432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1313.198254] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1319.744545] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.082519] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1331.264227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1331.839927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.997491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1338.064842] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1344.463216] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1351.369617] Lustre: lustre-OST0000: Not available for connect from 192.168.201.48@tcp (stopping) [ 1353.710061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.711184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.716933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.731248] Lustre: Skipped 1 previous similar message [ 1356.467734] Lustre: lustre-OST0000: Not available for connect from 192.168.201.48@tcp (stopping) [ 1356.482094] Lustre: Skipped 1 previous similar message [ 1356.753817] Lustre: server umount lustre-OST0000 complete [ 1379.306328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.312806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1379.313241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.335039] Lustre: Skipped 1 previous similar message [ 1379.388370] LustreError: Skipped 1 previous similar message [ 1383.539528] Lustre: server umount lustre-MDT0000 complete [ 1384.422387] LustreError: 38011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.436135] LustreError: 38011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1387.044137] LustreError: 37991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772435403 with bad export cookie 761634295099013714 [ 1387.056225] LustreError: 37991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1387.065394] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.475817] Lustre: server umount lustre-MDT0001 complete [ 1407.144094] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1410.374534] Key type lgssc unregistered [ 1410.771801] LNet: 41368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.787854] LNetError: 41368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.802800] LNet: Removed LNI 192.168.201.148@tcp [ 1411.832688] Key type .llcrypt unregistered [ 1411.834814] Key type ._llcrypt unregistered [ 1438.015440] Key type ._llcrypt registered [ 1438.018761] Key type .llcrypt registered [ 1438.196687] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1439.863628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.892784] alg: No test for adler32 (adler32-zlib) [ 1440.994534] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1441.289881] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1443.033233] Key type lgssc registered [ 1444.055248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1452.067426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1452.105190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.644287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.375510] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1463.789899] LustreError: 42766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.816110] LustreError: 42766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1468.577081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.898343] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1469.031919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.088486] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1480.716677] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.439831] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1492.307137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.758513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.943211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1499.949864] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1505.513466] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1513.160136] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.48@tcp (stopping) [ 1514.065192] Lustre: server umount lustre-MDT0000 complete [ 1514.477604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.481385] LustreError: 44381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.489347] Lustre: Skipped 2 previous similar messages [ 1514.518425] LustreError: 44381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1517.981233] LustreError: 42747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772435534 with bad export cookie 16910578635846860079 [ 1517.997860] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.001599] LustreError: 42747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1518.255976] LustreError: 42761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.312960] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [ 1519.594182] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.592948] LustreError: 42762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.601733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1522.613567] LustreError: 42762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1522.633772] Lustre: Skipped 1 previous similar message [ 1527.718198] LustreError: 42763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.720123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1527.743233] LustreError: 42763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1527.772709] Lustre: Skipped 1 previous similar message [ 1532.385229] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1532.758029] Lustre: server umount lustre-MDT0001 complete [ 1549.070042] Lustre: server umount lustre-OST0000 complete [ 1567.237948] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1570.903499] Key type lgssc unregistered [ 1571.217436] LNet: 46278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.224670] LNetError: 46278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.243675] LNet: Removed LNI 192.168.201.148@tcp [ 1572.493136] Key type .llcrypt unregistered [ 1572.494780] Key type ._llcrypt unregistered [ 1600.398796] Key type ._llcrypt registered [ 1600.400463] Key type .llcrypt registered [ 1600.546645] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1601.421612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.471282] alg: No test for adler32 (adler32-zlib) [ 1602.671633] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1602.841479] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1604.496236] Key type lgssc registered [ 1606.316850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.121582] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.164910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.884903] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1636.000277] LustreError: 47661:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 1662.752708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.006775] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1674.641855] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1839.332945] Lustre: server umount lustre-OST0000 complete [ 1854.447462] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1857.057390] Key type lgssc unregistered [ 1857.409109] LNet: 49206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.423056] LNetError: 49206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.454863] LNet: Removed LNI 192.168.201.148@tcp [ 1858.469401] Key type .llcrypt unregistered [ 1858.475302] Key type ._llcrypt unregistered [ 1884.463473] Key type ._llcrypt registered [ 1884.466506] Key type .llcrypt registered [ 1884.637860] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1885.876200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.065596] alg: No test for adler32 (adler32-zlib) [ 1887.086570] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1887.284101] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1888.928201] Key type lgssc registered [ 1889.892359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1897.639464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1897.664102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.128629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.231294] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1908.712441] LustreError: 50605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.728443] LustreError: 50605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1911.025186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.350649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.012157] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1921.597800] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1925.235197] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1932.808910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.186807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.876361] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1944.451727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1945.817815] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1949.671135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.674716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.685457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.701373] Lustre: Skipped 1 previous similar message [ 1950.507539] Lustre: server umount lustre-OST0000 complete [ 1954.787141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.793484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.803863] Lustre: Skipped 2 previous similar messages [ 1954.831105] Lustre: Skipped 1 previous similar message [ 1959.909148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.917656] Lustre: Skipped 1 previous similar message [ 1960.243851] Lustre: server umount lustre-MDT0000 complete [ 1963.823454] LustreError: 50583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772435980 with bad export cookie 16175629790413909529 [ 1963.827523] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.835330] LustreError: 50583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1964.104449] Lustre: server umount lustre-MDT0001 complete [ 1974.131377] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1977.066652] Key type lgssc unregistered [ 1977.345846] LNet: 53782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.354390] LNetError: 53782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.407423] LNet: Removed LNI 192.168.201.148@tcp [ 1979.144632] Key type .llcrypt unregistered [ 1979.146737] Key type ._llcrypt unregistered [ 2002.947510] Key type ._llcrypt registered [ 2002.949221] Key type .llcrypt registered [ 2003.141778] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2004.627155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.654835] alg: No test for adler32 (adler32-zlib) [ 2005.679758] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2005.902788] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2007.568218] Key type lgssc registered [ 2008.758921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.854818] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2016.878641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2032.416751] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2037.537061] LustreError: 55150:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 2064.224224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.907722] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2075.019125] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2081.554910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.253503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.323064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2087.076102] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2087.402939] LustreError: 55175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.431200] LustreError: 55175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2092.518445] LustreError: 55174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.424816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.746270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2101.059180] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2105.570413] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2109.008873] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2113.003461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.006732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.009369] LustreError: Skipped 1 previous similar message [ 2113.014817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.030960] Lustre: Skipped 1 previous similar message [ 2113.043465] Lustre: Skipped 1 previous similar message [ 2114.537643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.652406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.659751] Lustre: Skipped 2 previous similar messages [ 2124.771780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2124.784725] Lustre: Skipped 1 previous similar message [ 2125.280302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2125.440445] Lustre: server umount lustre-OST0000 complete [ 2148.836636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.844802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.865791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.374756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.451982] Lustre: server umount lustre-MDT0000 complete [ 2154.467719] LustreError: 57429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.497866] LustreError: 57429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2155.365351] LustreError: 55157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436171 with bad export cookie 6849009896699395895 [ 2155.373276] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.380367] LustreError: 55157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2161.812167] Lustre: server umount lustre-MDT0001 complete [ 2170.697888] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2172.626438] Key type lgssc unregistered [ 2172.821564] LNet: 58542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.826092] LNetError: 58542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.835528] LNet: Removed LNI 192.168.201.148@tcp [ 2173.432799] Key type .llcrypt unregistered [ 2173.434792] Key type ._llcrypt unregistered [ 2189.983495] Key type ._llcrypt registered [ 2189.985466] Key type .llcrypt registered [ 2190.059616] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2190.761393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2190.779381] alg: No test for adler32 (adler32-zlib) [ 2191.720906] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2191.850739] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2193.472563] Key type lgssc registered [ 2194.138373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.846959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.859075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.159297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.265319] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2211.299073] LustreError: 59917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.319082] LustreError: 59917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2211.615529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.923272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.868718] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2220.403964] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2223.590671] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2229.018202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2229.234244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.044841] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2238.873745] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2245.618416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2255.850078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2255.859994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.866065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2255.868966] Lustre: Skipped 1 previous similar message [ 2260.829328] Lustre: server umount lustre-OST0000 complete [ 2266.081919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.100273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.104470] Lustre: Skipped 1 previous similar message [ 2270.016568] Lustre: server umount lustre-MDT0000 complete [ 2271.203239] LustreError: 59916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.232701] LustreError: 59916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2273.127586] LustreError: 59898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436289 with bad export cookie 8542822732165976514 [ 2273.134969] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.141282] LustreError: 59898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2273.384450] Lustre: server umount lustre-MDT0001 complete [ 2281.643131] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2283.660638] Key type lgssc unregistered [ 2283.902529] LNet: 63116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.909683] LNetError: 63116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.924353] LNet: Removed LNI 192.168.201.148@tcp [ 2284.603145] Key type .llcrypt unregistered [ 2284.605428] Key type ._llcrypt unregistered [ 2303.443670] Key type ._llcrypt registered [ 2303.445627] Key type .llcrypt registered [ 2303.521404] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2304.315570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2304.333187] alg: No test for adler32 (adler32-zlib) [ 2305.300316] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2305.450013] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2307.136164] Key type lgssc registered [ 2307.928595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.508660] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:25:30 (1772436330) [ 2320.210167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2320.249306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.746611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.604748] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2332.136439] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.167712] LustreError: 64785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2332.695924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.974710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.938773] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2341.347665] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2344.673893] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2350.122515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2350.418030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.177819] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2359.689376] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2366.847828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2373.858188] Lustre: Failing over lustre-MDT0000 [ 2374.113217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.116110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.137669] LustreError: 64784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.361565] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.077238] Lustre: server umount lustre-MDT0000 complete [ 2377.193743] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.207642] Lustre: Skipped 1 previous similar message [ 2380.457868] LustreError: 64779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.473283] LustreError: 64779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2383.892374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.071532] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.183219] Lustre: *** cfs_fail_loc=135, val=0*** [ 2384.257949] LustreError: 67554:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 2384.261990] LustreError: 67554:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -2 [ 2384.267502] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2384.267502] [ 2384.274648] LustreError: MGC192.168.201.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2384.282693] LustreError: 67543:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2384.287232] LustreError: 67543:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 2384.292342] LustreError: 67543:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2384.297907] LustreError: 67543:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2384.311723] Lustre: server umount lustre-MDT0000 complete [ 2384.313965] LustreError: 67543:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2385.583047] LustreError: 66879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.596115] LustreError: 66879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2389.554762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.638908] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.647743] LustreError: 67899:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.201.148@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000002e8ad0c2) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2389.661505] LustreError: 64776:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2389.885906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.933455] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2390.698121] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2393.157977] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2395.122212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2395.140909] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2395.171848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2398.695098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2398.700413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.707685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.230021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.236545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.690802] Lustre: server umount lustre-OST0000 complete [ 2410.466903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.468014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2410.470160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.470168] Lustre: Skipped 1 previous similar message [ 2410.473223] Lustre: Skipped 2 previous similar messages [ 2413.838577] Lustre: server umount lustre-MDT0000 complete [ 2415.593721] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.608446] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2417.090106] LustreError: 66398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436433 with bad export cookie 6031380547869727418 [ 2417.098994] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.303711] Lustre: server umount lustre-MDT0001 complete [ 2424.757518] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2426.782779] Key type lgssc unregistered [ 2427.038767] LNet: 69301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.044142] LNetError: 69301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2427.076590] LNet: Removed LNI 192.168.201.148@tcp [ 2427.628272] Key type .llcrypt unregistered [ 2427.630417] Key type ._llcrypt unregistered [ 2443.355675] Key type ._llcrypt registered [ 2443.359293] Key type .llcrypt registered [ 2443.442670] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2444.009949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.041420] alg: No test for adler32 (adler32-zlib) [ 2445.053699] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2445.194252] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2446.841517] Key type lgssc registered [ 2447.557646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2453.204947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.221843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.568642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.236534] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2464.740149] LustreError: 70676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.754027] LustreError: 70676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2465.012472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.338472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2468.551774] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2472.785973] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2475.569513] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2480.709131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2480.916235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.069592] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2489.244615] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2500.400693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2504.924913] Lustre: Failing over lustre-MDT0000 [ 2505.122351] Lustre: server umount lustre-MDT0000 complete [ 2505.701032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.711506] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.721394] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2510.829737] LustreError: 71386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.844248] LustreError: 71386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2511.778663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.861753] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.984148] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2512.079629] LustreError: 73445:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-9) [ 2512.084420] LustreError: 73445:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -9 [ 2512.091428] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2512.091428] [ 2512.101538] LustreError: MGC192.168.201.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2512.112131] LustreError: 73433:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2512.118279] LustreError: 73433:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -9 [ 2512.125298] LustreError: 73433:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2512.133991] LustreError: 73433:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2512.153579] Lustre: server umount lustre-MDT0000 complete [ 2512.156383] LustreError: 73433:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2515.938916] LustreError: 70670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.951936] LustreError: 70670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2517.246608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.367581] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.658346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.734474] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2520.991797] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2521.776864] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2523.120889] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2523.156821] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2523.196870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2528.228113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.231795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.245565] Lustre: Skipped 3 previous similar messages [ 2528.256756] Lustre: Skipped 1 previous similar message [ 2532.921441] Lustre: server umount lustre-OST0000 complete [ 2538.465618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.473184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2538.485223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.502091] Lustre: Skipped 1 previous similar message [ 2543.585631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.593071] Lustre: Skipped 1 previous similar message [ 2547.687273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.691824] Lustre: Skipped 1 previous similar message [ 2550.241133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2550.465546] Lustre: server umount lustre-MDT0000 complete [ 2552.809749] LustreError: 70671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.834118] LustreError: 70671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2553.461620] LustreError: 70657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436570 with bad export cookie 15992861208445608419 [ 2553.467477] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.472469] LustreError: 70657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2553.848559] Lustre: server umount lustre-MDT0001 complete [ 2562.029331] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2564.323931] Key type lgssc unregistered [ 2564.609591] LNet: 75196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.616003] LNetError: 75196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2564.628518] LNet: Removed LNI 192.168.201.148@tcp [ 2565.264156] Key type .llcrypt unregistered [ 2565.268166] Key type ._llcrypt unregistered [ 2574.877509] Key type ._llcrypt registered [ 2574.879611] Key type .llcrypt registered [ 2575.001691] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2575.259317] Key type .llcrypt unregistered [ 2575.263473] Key type ._llcrypt unregistered [ 2592.962248] Key type ._llcrypt registered [ 2592.963772] Key type .llcrypt registered [ 2593.057742] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2593.842773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.862879] alg: No test for adler32 (adler32-zlib) [ 2594.861503] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2595.070419] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2596.696352] Key type lgssc registered [ 2597.622671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.669914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2603.683658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.066109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.272245] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2614.958410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.249912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2618.385348] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2622.276670] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2624.929281] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2630.278544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2630.520503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.544762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2634.825669] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2638.897977] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2651.111273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.112726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2651.116099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.121669] Lustre: Skipped 1 previous similar message [ 2653.315961] Lustre: server umount lustre-OST0000 complete [ 2656.225687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2656.228945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.232824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.243654] Lustre: Skipped 1 previous similar message [ 2656.247826] Lustre: Skipped 1 previous similar message [ 2661.346359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.351291] Lustre: Skipped 2 previous similar messages [ 2662.299374] Lustre: server umount lustre-MDT0000 complete [ 2665.153114] LustreError: 77027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436681 with bad export cookie 1180844021630348874 [ 2665.164821] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.167583] LustreError: 77027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2665.391525] Lustre: server umount lustre-MDT0001 complete [ 2672.201423] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2674.160890] Key type lgssc unregistered [ 2674.407155] LNet: 80224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.414255] LNetError: 80224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.422219] LNet: Removed LNI 192.168.201.148@tcp [ 2675.057607] Key type .llcrypt unregistered [ 2675.060102] Key type ._llcrypt unregistered [ 2689.081293] Key type ._llcrypt registered [ 2689.083444] Key type .llcrypt registered [ 2689.142365] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2689.797624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.836335] alg: No test for adler32 (adler32-zlib) [ 2690.782527] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2690.958347] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2692.584257] Key type lgssc registered [ 2693.352565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.829775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2698.845491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.278693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.262126] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2709.481185] LustreError: 81598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.502583] LustreError: 81598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2709.567928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.816805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.656394] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2716.227255] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.762423] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2723.412478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.692257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.726367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2727.906464] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2732.080372] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2740.193963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2740.196493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.199747] LustreError: Skipped 1 previous similar message [ 2740.202648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.218828] Lustre: Skipped 1 previous similar message [ 2745.317362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.325792] Lustre: Skipped 1 previous similar message [ 2746.377505] Lustre: server umount lustre-OST0000 complete [ 2750.436465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.446598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.449499] Lustre: Skipped 1 previous similar message [ 2750.460885] Lustre: Skipped 2 previous similar messages [ 2755.562877] LustreError: 81599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.577959] LustreError: 81599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2755.594364] Lustre: server umount lustre-MDT0000 complete [ 2757.399178] LustreError: 81579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436773 with bad export cookie 569423574364138588 [ 2757.408231] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.584484] Lustre: server umount lustre-MDT0001 complete [ 2763.344657] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2764.860911] Key type lgssc unregistered [ 2765.008467] LNet: 84776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2765.011904] LNetError: 84776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2765.025498] LNet: Removed LNI 192.168.201.148@tcp [ 2765.411901] Key type .llcrypt unregistered [ 2765.413565] Key type ._llcrypt unregistered [ 2779.569438] Key type ._llcrypt registered [ 2779.570580] Key type .llcrypt registered [ 2779.616168] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2780.019912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.068846] alg: No test for adler32 (adler32-zlib) [ 2780.982023] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2781.120525] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2782.744149] Key type lgssc registered [ 2783.295225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.712180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.724550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.003945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.133159] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2796.748312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.018025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2799.636736] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2803.344530] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2805.828568] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2809.907983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.150562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.415051] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2814.255349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2816.456148] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2829.798517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2829.804226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.812529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2832.866683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.868590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2832.890392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2834.973479] Lustre: server umount lustre-OST0000 complete [ 2837.985295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2837.988927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.990292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2837.990301] Lustre: Skipped 1 previous similar message [ 2838.000446] Lustre: Skipped 1 previous similar message [ 2843.106351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.111746] Lustre: Skipped 2 previous similar messages [ 2848.227443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.232232] Lustre: Skipped 1 previous similar message [ 2852.320157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2852.441672] Lustre: server umount lustre-MDT0000 complete [ 2853.346052] LustreError: 86150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.362563] LustreError: 86150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2855.211995] LustreError: 86131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436871 with bad export cookie 6164445715193361664 [ 2855.219850] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2855.227206] LustreError: 86131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2855.438929] Lustre: server umount lustre-MDT0001 complete [ 2864.524903] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2866.456791] Key type lgssc unregistered [ 2866.657777] LNet: 89358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.661941] LNetError: 89358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2866.676520] LNet: Removed LNI 192.168.201.148@tcp [ 2867.246121] Key type .llcrypt unregistered [ 2867.247658] Key type ._llcrypt unregistered [ 2882.956318] Key type ._llcrypt registered [ 2882.957838] Key type .llcrypt registered [ 2883.019229] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2883.675620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.709340] alg: No test for adler32 (adler32-zlib) [ 2884.690775] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2884.889418] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2886.520240] Key type lgssc registered [ 2887.351614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.544896] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2892.573286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.064678] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2913.184194] LustreError: 90705:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 2939.872240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.700243] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2948.947215] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2953.143455] Lustre: server umount lustre-OST0000 complete [ 2959.183098] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:36:15 (1772436975) [ 2964.126398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.393675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.271807] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2969.572269] LustreError: 92229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.587131] LustreError: 92229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2973.194271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.400088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2976.435800] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2980.831464] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2983.404189] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2988.064831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.304412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.391208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2992.575951] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2996.672632] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3005.921739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3005.926116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.937231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.970781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.972519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3007.987054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.389146] Lustre: server umount lustre-OST0000 complete [ 3014.627346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3014.636113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.647217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.650522] Lustre: Skipped 1 previous similar message [ 3018.209441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.210135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.227124] Lustre: Skipped 1 previous similar message [ 3020.718507] Lustre: server umount lustre-MDT0000 complete [ 3023.329479] LustreError: 92230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.344880] LustreError: 92230:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3023.728405] LustreError: 92210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437040 with bad export cookie 903533788490153304 [ 3023.734741] LustreError: 92210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3023.734784] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.009613] Lustre: server umount lustre-MDT0001 complete [ 3030.472621] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 3032.300427] Key type lgssc unregistered [ 3032.531584] LNet: 95566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3032.534851] LNetError: 95566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3032.542054] LNet: Removed LNI 192.168.201.148@tcp [ 3033.024486] Key type .llcrypt unregistered [ 3033.028090] Key type ._llcrypt unregistered [ 3048.620282] Key type ._llcrypt registered [ 3048.622388] Key type .llcrypt registered [ 3048.680878] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3049.398543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3049.459436] alg: No test for adler32 (adler32-zlib) [ 3050.415313] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3050.581917] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 3052.280717] Key type lgssc registered [ 3053.124813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3058.570641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3058.597323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.953575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.783517] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3068.149351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.363541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.694343] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3074.607271] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3077.580069] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3082.153621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.374139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.489767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3085.916165] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3089.390595] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3101.154526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3101.166397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.178360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.201451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3103.209428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.211098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.234708] Lustre: Skipped 1 previous similar message [ 3104.619112] Lustre: server umount lustre-OST0000 complete [ 3108.322487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.325897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.331217] Lustre: Skipped 1 previous similar message [ 3112.882775] Lustre: server umount lustre-MDT0000 complete [ 3113.441877] LustreError: 97649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.450147] LustreError: 97649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3115.342245] LustreError: 96919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437131 with bad export cookie 5129735685335536549 [ 3115.343726] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.347469] LustreError: 96919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3115.580107] Lustre: server umount lustre-MDT0001 complete [ 3122.081249] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 3123.866178] Key type lgssc unregistered [ 3124.058742] LNet: 100118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3124.067674] LNetError: 100118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3124.080367] LNet: Removed LNI 192.168.201.148@tcp [ 3124.597430] Key type .llcrypt unregistered [ 3124.599288] Key type ._llcrypt unregistered [ 3135.826122] Key type ._llcrypt registered [ 3135.827561] Key type .llcrypt registered [ 3135.908727] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3136.312743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3136.358677] alg: No test for adler32 (adler32-zlib) [ 3137.260247] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3137.379891] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 3138.993098] Key type lgssc registered [ 3139.752254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3144.797872] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3144.813560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.288498] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3165.411091] LustreError: 101322:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 3192.032201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.808079] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3199.174099] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3203.272170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.444227] LustreError: MGC192.168.201.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3203.453313] LustreError: 102233:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3203.462819] LustreError: 102233:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 3203.471037] LustreError: 102233:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 3203.536460] Lustre: server umount lustre-MDT0000 complete [ 3203.538631] LustreError: 102233:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3219.021137] Lustre: server umount lustre-OST0000 complete [ 3226.048162] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3232.078054] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3257.177745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.368306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3257.391380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3257.446920] Lustre: lustre-MDT0000: new disk, initializing [ 3257.502382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.516937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3260.406545] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3268.248569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.336324] Lustre: 104986:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3268.361301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3268.364466] Lustre: Skipped 1 previous similar message [ 3268.427648] Lustre: lustre-MDT0001: new disk, initializing [ 3268.503656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.542419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3268.553731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3271.397477] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3276.854752] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3279.451275] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3284.485160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3284.657577] Lustre: lustre-OST0000: new disk, initializing [ 3284.662044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3284.730292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.610505] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3290.711179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3290.724421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3290.827962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3295.417356] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3298.408576] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3298.530301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3301.625974] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3301.776856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3306.468162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.470693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.480836] LustreError: Skipped 1 previous similar message [ 3306.489636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3306.493844] Lustre: Skipped 1 previous similar message [ 3306.510048] Lustre: Skipped 1 previous similar message [ 3309.272918] Lustre: server umount lustre-OST0000 complete [ 3314.658375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.668774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.675244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.705515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.705672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.725532] Lustre: Skipped 1 previous similar message [ 3318.425815] Lustre: server umount lustre-MDT0000 complete [ 3321.104267] LustreError: 105765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437337 with bad export cookie 8898317677669105085 [ 3321.108831] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.112784] LustreError: 105765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3321.446047] Lustre: server umount lustre-MDT0001 complete [ 3327.527679] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:42:23 (1772437343) [ 3329.962666] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3341.068240] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3347.269979] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3373.672324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.947832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3373.979921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3374.039770] Lustre: lustre-MDT0000: new disk, initializing [ 3374.107152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.128564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3377.374670] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3385.859562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.946926] Lustre: 111442:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3385.997463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3386.000569] Lustre: Skipped 1 previous similar message [ 3386.088534] Lustre: lustre-MDT0001: new disk, initializing [ 3386.209973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3386.251201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3386.271266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3389.633444] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3395.566037] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.222555] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3402.919791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.152708] Lustre: lustre-OST0000: new disk, initializing [ 3403.156104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3403.217807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.560048] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3408.917245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3408.924584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3408.991910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3412.936413] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3416.002495] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3416.154204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3419.432542] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3419.672940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3424.230594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.232213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.235408] LustreError: Skipped 1 previous similar message [ 3424.242083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.247501] Lustre: Skipped 1 previous similar message [ 3427.345640] Lustre: server umount lustre-OST0000 complete [ 3432.416925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3432.420069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.425397] Lustre: Skipped 1 previous similar message [ 3432.427739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.682225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.686183] Lustre: Skipped 5 previous similar messages [ 3444.192352] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3444.284847] Lustre: server umount lustre-MDT0000 complete [ 3446.603716] LustreError: 112221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437463 with bad export cookie 8898317677669105967 [ 3446.606758] LustreError: 112221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3446.607132] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.832483] Lustre: server umount lustre-MDT0001 complete [ 3452.148464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.431089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.275927] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3457.505865] LustreError: 115251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.520153] LustreError: 115251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3460.757479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.479578] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3467.077680] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3469.428314] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3473.572591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.998125] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3480.001333] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3487.845503] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3491.300245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.300852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3491.310884] Lustre: Skipped 1 previous similar message [ 3491.311922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.326277] Lustre: Skipped 1 previous similar message [ 3495.521833] Lustre: server umount lustre-OST0000 complete [ 3503.737190] Lustre: server umount lustre-MDT0000 complete [ 3505.713300] LustreError: 115231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437522 with bad export cookie 8898317677669106849 [ 3505.716166] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.720231] LustreError: 115231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3505.862281] Lustre: server umount lustre-MDT0001 complete [ 3512.546989] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 3514.508911] Key type lgssc unregistered [ 3514.702901] LNet: 118524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3514.708440] LNetError: 118524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3514.721836] LNet: Removed LNI 192.168.201.148@tcp [ 3515.387494] Key type .llcrypt unregistered [ 3515.389474] Key type ._llcrypt unregistered [ 3526.223556] Key type ._llcrypt registered [ 3526.224529] Key type .llcrypt registered [ 3526.264526] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3535.285704] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3536.021530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3536.058060] alg: No test for adler32 (adler32-zlib) [ 3537.051121] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3537.204502] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 3538.824341] Key type lgssc registered [ 3539.583827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3557.409773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3557.452130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.645786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3558.675824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3558.735419] Lustre: lustre-MDT0000: new disk, initializing [ 3558.817454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.836261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.976757] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3566.743422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.800675] Lustre: 121616:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3566.851043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3566.857736] Lustre: Skipped 1 previous similar message [ 3566.898589] Lustre: lustre-MDT0001: new disk, initializing [ 3566.958902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3566.976135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3566.980992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3569.162321] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3573.266776] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3575.128216] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3578.744310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3578.924409] Lustre: lustre-OST0000: new disk, initializing [ 3578.927653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3578.988055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.491841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3580.495569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3580.546827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3581.778317] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3585.625845] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3587.636786] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3587.722299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3589.657755] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3589.745564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3591.139797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.140727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3591.147174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.151139] LustreError: Skipped 1 previous similar message [ 3591.156075] Lustre: Skipped 1 previous similar message [ 3596.262785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.764396] Lustre: server umount lustre-OST0000 complete [ 3601.378450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.392029] Lustre: Skipped 1 previous similar message [ 3601.396879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3601.400074] Lustre: Skipped 2 previous similar messages [ 3604.765981] Lustre: server umount lustre-MDT0000 complete [ 3606.514415] LustreError: 121610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437623 with bad export cookie 16107922576428791256 [ 3606.517840] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3606.520744] LustreError: 121610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3606.681628] Lustre: server umount lustre-MDT0001 complete [ 3610.807336] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:47:07 (1772437627) [ 3613.794363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.025098] LustreError: 125624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.035501] LustreError: 125624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3614.079303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.695385] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3619.110497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.295175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.940328] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3623.289305] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3624.833774] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3628.770357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.771625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3628.778145] Lustre: Skipped 2 previous similar messages [ 3631.840759] Lustre: server umount lustre-MDT0000 complete [ 3633.551293] LustreError: 125650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437650 with bad export cookie 16107922576428792138 [ 3633.552195] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.557057] LustreError: 125650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3633.712179] Lustre: server umount lustre-MDT0001 complete [ 3637.534159] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:47:33 (1772437653) [ 3640.978238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.480737] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3661.600409] LustreError: 127812:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 3687.200821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.621390] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3691.997916] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3693.054046] Lustre: server umount lustre-OST0000 complete [ 3696.820060] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:48:33 (1772437713) [ 3699.765930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.015704] LustreError: 129142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.796139] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3705.313548] LustreError: 129143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.443474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.695729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.701061] Lustre: Skipped 1 previous similar message [ 3708.703748] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3711.258268] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3713.174981] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3716.560284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.469399] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3722.161859] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.404206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.405142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3730.411849] Lustre: Skipped 2 previous similar messages [ 3730.415702] Lustre: Skipped 3 previous similar messages [ 3735.419458] Lustre: server umount lustre-MDT0000 complete [ 3735.522054] LustreError: 129142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.528807] LustreError: 129142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3737.107977] LustreError: 130760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437753 with bad export cookie 16107922576428792789 [ 3737.108578] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3737.114436] LustreError: 130760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3737.243986] Lustre: server umount lustre-MDT0001 complete [ 3749.174880] Lustre: server umount lustre-OST0000 complete [ 3753.152468] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:49:29 (1772437769) [ 3756.269520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.495779] LustreError: 132459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.543104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.546752] Lustre: Skipped 1 previous similar message [ 3758.394735] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3763.041826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.260907] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3768.315812] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3770.278683] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3773.777462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.462731] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3779.416324] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3781.549376] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3782.122881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3782.677240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3785.081880] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3789.326450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3792.354835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3792.356925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.363939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.380502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.409861] Lustre: server umount lustre-OST0000 complete [ 3799.010527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3803.618155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.622667] Lustre: Skipped 2 previous similar messages [ 3804.765807] Lustre: server umount lustre-MDT0000 complete [ 3806.211173] LustreError: 132439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437822 with bad export cookie 16107922576428793846 [ 3806.214164] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3806.218262] LustreError: 132439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3806.342782] Lustre: server umount lustre-MDT0001 complete [ 3809.809613] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:50:26 (1772437826) [ 3812.298361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.744708] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3832.864412] LustreError: 136278:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 3858.467101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.469251] Lustre: Skipped 2 previous similar messages [ 3861.239727] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3864.186764] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3867.551720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.736989] LustreError: 136303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.748387] LustreError: 136303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3867.837314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3869.626533] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3873.147402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.864349] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3877.177588] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3878.597484] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3880.282260] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3880.368214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3881.929206] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3882.029571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3883.490636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3883.491458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.499612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3883.501878] Lustre: Skipped 7 previous similar messages [ 3897.312107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3897.385541] Lustre: server umount lustre-MDT0000 complete [ 3898.852057] LustreError: 137874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.858068] LustreError: 137874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3898.988598] LustreError: 136285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437915 with bad export cookie 16107922576428794868 [ 3898.989471] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.993162] LustreError: 136285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3899.118841] Lustre: server umount lustre-MDT0001 complete [ 3911.077076] Lustre: server umount lustre-OST0000 complete [ 3914.512974] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:52:10 (1772437930) [ 3917.570945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.088736] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3938.208840] LustreError: 140100:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 3966.316851] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3968.699186] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3971.681543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.898062] LustreError: 140125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.958883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3973.575030] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3976.959538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.543237] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3980.663390] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3982.074709] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3984.588846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3984.664995] Lustre: lustre-OST0001: new disk, initializing [ 3984.667726] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3987.011972] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3990.427208] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3991.561236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3991.566082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3991.627083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3992.095089] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3992.176104] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3993.874304] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3993.968966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3996.643148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.648859] Lustre: Skipped 2 previous similar messages [ 3996.651037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3996.653588] Lustre: Skipped 7 previous similar messages [ 4000.907963] Lustre: server umount lustre-OST0000 complete [ 4006.883418] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.739812] Lustre: server umount lustre-OST0001 complete [ 4016.667690] Lustre: server umount lustre-MDT0000 complete [ 4018.117787] LustreError: 140106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438034 with bad export cookie 16107922576428795820 [ 4018.118862] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4018.129392] LustreError: 140106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4018.239667] Lustre: server umount lustre-MDT0001 complete [ 4027.393548] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:54:03 (1772438043) [ 4027.975799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4028.565397] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4029.164067] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:54:05 (1772438045) [ 4031.605947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.721482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4031.733337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4031.819693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.822645] Lustre: Skipped 6 previous similar messages [ 4033.274270] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4036.730063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.782330] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4036.792883] Lustre: 146043:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4038.535768] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4040.898996] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4042.343720] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4045.344873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.422286] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4047.888689] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4048.684991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4050.090557] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4051.932805] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4052.027572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4053.688410] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4053.764628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4054.652717] Lustre: server umount lustre-OST0000 complete [ 4069.042939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4069.175951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.182336] Lustre: Skipped 6 previous similar messages [ 4071.758622] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4074.403550] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4074.471244] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4074.476165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4074.477599] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4074.485209] Lustre: Skipped 1 previous similar message [ 4076.507450] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4076.604808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4078.290246] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4078.384460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4079.903927] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96fe4c131800.ost_server_uuid 50 [ 4080.603834] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96fe4c131800.ost_server_uuid in FULL state after 0 sec [ 4080.685654] LustreError: 146050:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.48@tcp) returned error from blocking AST (req@00000000a031b537 x1858535479769216 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f9c48bd9/0xdf8ad63d65889106 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.48@tcp remote: 0x31b0792e1ef30c26 expref: 5 pid: 147684 timeout: 4180 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4080.700108] LustreError: lustre-MDT0000: A client on nid 192.168.201.48@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4080.704471] LustreError: 146038:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.48@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f9c48bd9/0xdf8ad63d65889106 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.48@tcp remote: 0x31b0792e1ef30c26 expref: 6 pid: 147684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4084.707705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.708464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.710201] LustreError: Skipped 2 previous similar messages [ 4084.715820] Lustre: Skipped 7 previous similar messages [ 4088.164804] Lustre: server umount lustre-OST0000 complete [ 4096.162131] Lustre: server umount lustre-MDT0000 complete [ 4097.876148] LustreError: 146034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438114 with bad export cookie 16107922576428797017 [ 4097.879264] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.881187] LustreError: 146034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4112.352390] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4112.445791] Lustre: server umount lustre-MDT0001 complete [ 4116.597921] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4117.780826] Key type lgssc unregistered [ 4117.927459] LNet: 151517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4117.930790] LNetError: 151517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4117.941697] LNet: Removed LNI 192.168.201.148@tcp [ 4118.317788] Key type .llcrypt unregistered [ 4118.319156] Key type ._llcrypt unregistered [ 4128.469989] Key type ._llcrypt registered [ 4128.471452] Key type .llcrypt registered [ 4128.514061] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4128.866717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.881569] alg: No test for adler32 (adler32-zlib) [ 4129.749284] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4129.846024] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4131.440167] Key type lgssc registered [ 4131.927295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4135.249864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4135.257890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.450095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.106121] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4141.931922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.124404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.741858] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4145.933847] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4147.390132] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4150.263200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.423655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.661802] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4155.012381] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4162.726250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4169.451444] Lustre: Failing over lustre-MDT0000 [ 4169.607764] Lustre: server umount lustre-MDT0000 complete [ 4172.769707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4172.773922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.780035] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.786277] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4173.281548] LustreError: 152890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.286959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.291794] LustreError: 152890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4173.297013] Lustre: Skipped 1 previous similar message [ 4178.402417] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.409635] LustreError: 152891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4180.659965] LustreError: 153608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.48@tcp arrived at 1772438197 with bad export cookie 15868048017067306575 [ 4180.664257] LustreError: 153608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4183.522379] LustreError: 153614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.528554] LustreError: 153614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4183.633631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.686284] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4183.800795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.828772] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4185.329927] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4186.228692] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4186.232620] Lustre: lustre-MDT0000: Denying connection for new client 72322998-b450-4f40-97e6-547f7ea9a514 (at 192.168.201.48@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 4189.163458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4194.275141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.275996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.279559] Lustre: Skipped 1 previous similar message [ 4199.140785] Lustre: server umount lustre-OST0000 complete [ 4200.725893] LustreError: 156289:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4200.730415] Lustre: 155653:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4200.734543] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4200.742253] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4200.748729] LustreError: 155653:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000000412d5d x1858536101416576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4200.761818] LustreError: 155653:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4200.765942] LustreError: 155653:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4200.770189] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 4200.877711] Lustre: server umount lustre-MDT0000 complete [ 4202.565857] LustreError: 152872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438219 with bad export cookie 15868048017067306904 [ 4202.569679] LustreError: 152200:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4202.569832] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.571328] LustreError: 152872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4208.757418] Lustre: server umount lustre-MDT0001 complete [ 4212.698243] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4213.804951] Key type lgssc unregistered [ 4213.936317] LNet: 157024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4213.939258] LNetError: 157024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4213.951328] LNet: Removed LNI 192.168.201.148@tcp [ 4214.272196] Key type .llcrypt unregistered [ 4214.273297] Key type ._llcrypt unregistered [ 4223.922027] Key type ._llcrypt registered [ 4223.923706] Key type .llcrypt registered [ 4223.966346] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4224.306805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4224.347224] alg: No test for adler32 (adler32-zlib) [ 4225.228631] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4225.327437] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4226.920287] Key type lgssc registered [ 4227.339531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4230.194170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4230.203762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.443964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.851942] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4236.043972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.192799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4237.677460] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4239.812843] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4241.329037] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4243.871034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.015963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.164046] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4248.161719] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4257.320828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4261.345090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4261.347826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.353402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4262.371772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4262.372275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.372505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4266.966192] Lustre: server umount lustre-OST0000 complete [ 4272.097222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4272.100658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.106568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4272.108938] Lustre: Skipped 1 previous similar message [ 4274.534841] Lustre: server umount lustre-MDT0000 complete [ 4275.913255] LustreError: 158375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438292 with bad export cookie 7327609550527657787 [ 4275.914540] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4275.919258] LustreError: 158375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4276.050791] Lustre: server umount lustre-MDT0001 complete [ 4279.997582] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4281.185408] Key type lgssc unregistered [ 4281.319392] LNet: 161593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4281.322310] LNetError: 161593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4281.329477] LNet: Removed LNI 192.168.201.148@tcp [ 4281.675871] Key type .llcrypt unregistered [ 4281.677567] Key type ._llcrypt unregistered [ 4294.737503] Key type ._llcrypt registered [ 4294.739272] Key type .llcrypt registered [ 4294.789153] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4295.154814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4295.186040] alg: No test for adler32 (adler32-zlib) [ 4296.051615] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4296.140343] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4297.728122] Key type lgssc registered [ 4298.129938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4301.192719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4301.198138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.380921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.847444] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4306.855046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.001951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4308.370690] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4310.469731] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4312.028631] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4314.552448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.686108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.938079] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4319.029117] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4328.038026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4332.513272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4332.516250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.520674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4333.537527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4333.539565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.540195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.172972] Lustre: server umount lustre-OST0000 complete [ 4338.657031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.657832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.661695] Lustre: Skipped 1 previous similar message [ 4338.663923] Lustre: Skipped 1 previous similar message [ 4343.779491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.782980] Lustre: Skipped 2 previous similar messages [ 4344.805505] Lustre: server umount lustre-MDT0000 complete [ 4346.188439] LustreError: 163236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438362 with bad export cookie 17228919096719224344 [ 4346.191803] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.192668] LustreError: 163236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4346.324916] Lustre: server umount lustre-MDT0001 complete [ 4350.320911] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4351.540584] Key type lgssc unregistered [ 4351.687412] LNet: 166452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4351.690834] LNetError: 166452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4351.704315] LNet: Removed LNI 192.168.201.148@tcp [ 4352.060723] Key type .llcrypt unregistered [ 4352.062338] Key type ._llcrypt unregistered [ 4360.955807] Key type ._llcrypt registered [ 4360.956766] Key type .llcrypt registered [ 4360.993669] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4361.440560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4361.452855] alg: No test for adler32 (adler32-zlib) [ 4362.331851] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4362.429887] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4364.016158] Key type lgssc registered [ 4364.447674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4368.433432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4368.444114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.601903] Lustre: *** cfs_fail_loc=135, val=0*** [ 4369.643973] LustreError: 167937:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 4369.647084] LustreError: 167937:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -2 [ 4369.650047] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4369.650047] [ 4369.653555] LustreError: MGC192.168.201.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4369.657840] LustreError: 167898:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4369.661042] LustreError: 167898:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4369.663341] LustreError: 167898:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 4369.665413] LustreError: 167898:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 4369.705027] Lustre: server umount lustre-MDT0000 complete [ 4369.707445] LustreError: 167898:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4372.128984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4372.302024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4373.672787] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4376.683092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.825359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.233859] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4380.129725] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4381.538378] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4387.297639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4387.298244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.299352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4387.308220] Lustre: Skipped 1 previous similar message [ 4388.321346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.323839] Lustre: Skipped 2 previous similar messages [ 4388.443829] Lustre: server umount lustre-MDT0000 complete [ 4389.792143] LustreError: 168291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438406 with bad export cookie 50631013539174105 [ 4389.795409] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.796220] LustreError: 168291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4389.915827] Lustre: server umount lustre-MDT0001 complete [ 4393.821724] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4394.874335] Key type lgssc unregistered [ 4395.016638] LNet: 170383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4395.019405] LNetError: 170383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4395.030490] LNet: Removed LNI 192.168.201.148@tcp [ 4395.333860] Key type .llcrypt unregistered [ 4395.335040] Key type ._llcrypt unregistered [ 4403.280818] Key type ._llcrypt registered [ 4403.281739] Key type .llcrypt registered [ 4403.319340] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4403.416270] Key type .llcrypt unregistered [ 4403.417490] Key type ._llcrypt unregistered [ 4410.331579] Key type ._llcrypt registered [ 4410.332593] Key type .llcrypt registered [ 4410.367220] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4410.748228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4410.773080] alg: No test for adler32 (adler32-zlib) [ 4411.635852] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4411.718315] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4413.296166] Key type lgssc registered [ 4413.672254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4416.247922] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4416.253725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4431.713590] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4436.832225] LustreError: 172456:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 4463.456217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.621153] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4467.661296] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4470.218777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.415798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.437136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4471.834358] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4474.995545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.137298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4476.492855] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4478.610459] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4480.220274] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4481.374449] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4488.674824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.676968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.679345] Lustre: Skipped 1 previous similar message [ 4488.682779] Lustre: Skipped 1 previous similar message [ 4492.826351] Lustre: server umount lustre-OST0000 complete [ 4495.840972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4495.844689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4495.850144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.913123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.913563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.329977] Lustre: server umount lustre-MDT0000 complete [ 4501.830299] LustreError: 174101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438518 with bad export cookie 11465280703512209318 [ 4501.833269] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.834202] LustreError: 174101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4501.969199] Lustre: server umount lustre-MDT0001 complete [ 4505.988175] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4507.107925] Key type lgssc unregistered [ 4507.251464] LNet: 176030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4507.255196] LNetError: 176030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4507.268853] LNet: Removed LNI 192.168.201.148@tcp [ 4507.584569] Key type .llcrypt unregistered [ 4507.585797] Key type ._llcrypt unregistered [ 4517.368562] Key type ._llcrypt registered [ 4517.369791] Key type .llcrypt registered [ 4517.409121] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4517.756737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4517.806106] alg: No test for adler32 (adler32-zlib) [ 4518.663276] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4518.746738] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4520.328133] Key type lgssc registered [ 4520.732508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4523.488587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4523.496095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.658373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.086759] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4529.296163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.427421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4530.857381] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4532.873332] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4534.273401] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4536.789053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.912204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.903237] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4540.847926] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4550.245853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4553.600945] Lustre: Failing over lustre-MDT0000 [ 4553.695451] Lustre: server umount lustre-MDT0000 complete [ 4555.232755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4555.235761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.241135] LustreError: 177400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.247054] LustreError: 177400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4555.746323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.747069] LustreError: 177396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.751442] Lustre: Skipped 1 previous similar message [ 4555.758215] LustreError: 177396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4557.491401] LustreError: 177396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.865237] LustreError: 177401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.871395] LustreError: 177401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4565.985326] LustreError: 179554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.990350] LustreError: 179554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4566.698788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.741961] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.831155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.849223] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4567.721326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4568.154170] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4569.833856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4572.137040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4572.144531] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4572.160710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4574.660613] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4585.441112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4585.444034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.449576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4587.493948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4587.494429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.494601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4591.321383] Lustre: server umount lustre-OST0000 complete [ 4597.731807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.732083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4597.732499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4597.732504] Lustre: Skipped 1 previous similar message [ 4597.738889] Lustre: Skipped 1 previous similar message [ 4598.872063] Lustre: server umount lustre-MDT0000 complete [ 4600.345386] LustreError: 180892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438616 with bad export cookie 1889035506726915790 [ 4600.347370] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.349965] LustreError: 180892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4600.476291] Lustre: server umount lustre-MDT0001 complete [ 4604.472565] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4605.515267] Key type lgssc unregistered [ 4605.653343] LNet: 181933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4605.656887] LNetError: 181933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4605.667306] LNet: Removed LNI 192.168.201.148@tcp [ 4605.972758] Key type .llcrypt unregistered [ 4605.974435] Key type ._llcrypt unregistered [ 4615.476686] Key type ._llcrypt registered [ 4615.477799] Key type .llcrypt registered [ 4615.513110] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4615.934330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4615.944238] alg: No test for adler32 (adler32-zlib) [ 4616.805253] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4616.894540] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4618.480161] Key type lgssc registered [ 4618.847901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4621.496545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4621.502064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.678350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.105598] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4627.175947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.312829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.740439] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4630.623521] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4631.888447] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4634.207918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.323973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.312592] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4638.132184] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4648.681605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4650.311167] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4651.748932] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4658.804269] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4669.410252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4669.411081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.413500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.418031] Lustre: Skipped 1 previous similar message [ 4674.530926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.533467] Lustre: Skipped 2 previous similar messages [ 4675.153813] Lustre: server umount lustre-OST0000 complete [ 4678.624803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4678.627493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.631909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4679.649704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.582893] Lustre: server umount lustre-MDT0000 complete [ 4683.919070] LustreError: 184922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438700 with bad export cookie 1860164017441029903 [ 4683.921041] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4683.922667] LustreError: 184922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4684.038779] Lustre: server umount lustre-MDT0001 complete [ 4687.734120] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4688.735462] Key type lgssc unregistered [ 4688.869410] LNet: 186671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4688.872944] LNetError: 186671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4688.885323] LNet: Removed LNI 192.168.201.148@tcp [ 4689.160474] Key type .llcrypt unregistered [ 4689.161543] Key type ._llcrypt unregistered [ 4698.041690] Key type ._llcrypt registered [ 4698.042850] Key type .llcrypt registered [ 4698.080099] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4698.399825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4698.432057] alg: No test for adler32 (adler32-zlib) [ 4699.286266] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4699.365080] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4700.944177] Key type lgssc registered [ 4701.288446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4703.822978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4703.829193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4704.981197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.291121] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4709.228817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.363465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4710.715897] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4712.494530] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4713.661894] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4715.849484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.986602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.788204] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4719.565543] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4730.346368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4736.491868] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4745.296489] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4751.889251] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4762.059713] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4770.272705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4770.275173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4770.278907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4771.301133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4771.301453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.301612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4775.753762] Lustre: server umount lustre-OST0000 complete [ 4781.024786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4781.027237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.031345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4781.032943] Lustre: Skipped 1 previous similar message [ 4786.657243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.659415] Lustre: Skipped 2 previous similar messages [ 4791.264116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4791.319721] Lustre: server umount lustre-MDT0000 complete [ 4791.777021] LustreError: 188754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.784038] LustreError: 188754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4792.493333] LustreError: 188024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438809 with bad export cookie 13785533547212705634 [ 4792.496618] LustreError: 188024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4792.497166] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4792.603698] Lustre: server umount lustre-MDT0001 complete [ 4796.192716] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4797.187687] Key type lgssc unregistered [ 4797.313254] LNet: 193207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4797.315307] LNetError: 193207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4797.325284] LNet: Removed LNI 192.168.201.148@tcp [ 4797.619748] Key type .llcrypt unregistered [ 4797.620708] Key type ._llcrypt unregistered [ 4806.342569] Key type ._llcrypt registered [ 4806.343509] Key type .llcrypt registered [ 4806.385612] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4806.711517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4806.793638] alg: No test for adler32 (adler32-zlib) [ 4807.648752] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4807.729246] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4809.312117] Key type lgssc registered [ 4809.669631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4812.199689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4812.206609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.365909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.637441] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4817.367462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4817.503426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4818.804709] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4820.622866] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4821.852776] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4824.029567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.150637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.954516] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4827.768132] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4837.478622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4844.001460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.003842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.010616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4848.099916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4848.100082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.100472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4853.220981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4853.223215] Lustre: Skipped 2 previous similar messages [ 4857.312130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4857.371174] Lustre: server umount lustre-OST0000 complete [ 4863.457127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4863.458534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.461538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4863.464860] Lustre: Skipped 1 previous similar message [ 4864.797330] Lustre: server umount lustre-MDT0000 complete [ 4866.196350] LustreError: 194560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438882 with bad export cookie 10039932255399392175 [ 4866.198324] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.200034] LustreError: 194560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4866.325450] Lustre: server umount lustre-MDT0001 complete [ 4870.039207] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4871.098183] Key type lgssc unregistered [ 4871.230341] LNet: 197928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4871.233674] LNetError: 197928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4871.243342] LNet: Removed LNI 192.168.201.148@tcp [ 4871.563858] Key type .llcrypt unregistered [ 4871.565219] Key type ._llcrypt unregistered [ 4879.230388] Key type ._llcrypt registered [ 4879.232110] Key type .llcrypt registered [ 4879.273891] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4879.611731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4879.658241] alg: No test for adler32 (adler32-zlib) [ 4880.518018] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4880.603914] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4882.192159] Key type lgssc registered [ 4882.534314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4885.099985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4885.106329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.278067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.610756] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4890.352304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.488536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4891.778046] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4893.539552] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4894.710391] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4896.999294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4897.120349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.990721] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4900.757649] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4910.439415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4915.682803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.683063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.684563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.692209] Lustre: Skipped 1 previous similar message [ 4916.687950] Lustre: server umount lustre-OST0000 complete [ 4919.957446] Lustre: server umount lustre-MDT0000 complete [ 4920.801408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.802327] LustreError: 200514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.805021] Lustre: Skipped 1 previous similar message [ 4920.809882] LustreError: 200514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4921.158327] LustreError: 201479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438937 with bad export cookie 2417028350092498289 [ 4921.161536] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.163217] LustreError: 201479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4921.283672] Lustre: server umount lustre-MDT0001 complete [ 4924.734396] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4925.712846] Key type lgssc unregistered [ 4925.836385] LNet: 202501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4925.840079] LNetError: 202501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4925.846268] LNet: Removed LNI 192.168.201.148@tcp [ 4926.114736] Key type .llcrypt unregistered [ 4926.116122] Key type ._llcrypt unregistered [ 4933.189784] Key type ._llcrypt registered [ 4933.191123] Key type .llcrypt registered [ 4933.228332] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4933.667269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4933.673932] alg: No test for adler32 (adler32-zlib) [ 4934.533598] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4934.614126] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4936.200130] Key type lgssc registered [ 4936.570456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4939.126413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4939.132713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.308532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.646088] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4944.583985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.719332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4946.168141] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4948.018925] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4949.302223] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4951.726807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4951.862737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.828499] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4955.789680] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4965.159880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4968.959945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4968.984070] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4969.061722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.978667] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4972.680913] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4973.232748] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4983.762798] Lustre: Permanently deactivating lustre-OST0001 [ 4983.769100] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4991.479971] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4995.796476] Lustre: Permanently reactivating lustre-OST0001 [ 4995.802399] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4995.804778] Lustre: Skipped 2 previous similar messages [ 5000.184561] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5000.191671] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5000.195335] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5000.200463] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5000.212524] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5005.281501] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5005.281557] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.284142] LustreError: Skipped 1 previous similar message [ 5005.284758] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5005.287997] Lustre: Skipped 2 previous similar messages [ 5005.293037] Lustre: Skipped 1 previous similar message [ 5010.918457] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5010.921099] Lustre: Skipped 1 previous similar message [ 5011.135138] Lustre: server umount lustre-OST0001 complete [ 5016.035111] LustreError: 205317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.036488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.037911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5016.042414] LustreError: 205317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5016.051761] Lustre: Skipped 1 previous similar message [ 5018.516193] Lustre: server umount lustre-OST0000 complete [ 5021.153791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.154673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5021.157399] Lustre: Skipped 1 previous similar message [ 5021.160801] Lustre: Skipped 2 previous similar messages [ 5026.010578] Lustre: server umount lustre-MDT0000 complete [ 5026.273031] LustreError: 203727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.278338] LustreError: 203727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5027.271290] LustreError: 207091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439043 with bad export cookie 923560826827310167 [ 5027.274126] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.274985] LustreError: 207091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5027.397932] Lustre: server umount lustre-MDT0001 complete [ 5030.933685] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5031.999370] Key type lgssc unregistered [ 5032.129384] LNet: 208336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5032.131825] LNetError: 208336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5032.140358] LNet: Removed LNI 192.168.201.148@tcp [ 5032.443483] Key type .llcrypt unregistered [ 5032.445179] Key type ._llcrypt unregistered [ 5046.549627] Key type ._llcrypt registered [ 5046.550726] Key type .llcrypt registered [ 5046.589395] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5047.002121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5047.017250] alg: No test for adler32 (adler32-zlib) [ 5047.883923] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5047.977654] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5049.568126] Key type lgssc registered [ 5049.939402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5052.460266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5052.465614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.563483] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5053.572567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5053.642991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.993504] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5057.802877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.852368] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5057.870625] Lustre: 210497:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.973154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5059.274277] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5061.094750] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5062.272302] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5064.377664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.444429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5064.513545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.260761] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5068.007408] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5072.404380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5075.028675] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5085.841495] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5093.395375] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5101.984497] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5118.111531] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5118.114183] Lustre: Skipped 1 previous similar message [ 5140.704679] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5140.706635] Lustre: Skipped 2 previous similar messages [ 5176.343565] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5176.346397] Lustre: Skipped 4 previous similar messages [ 5236.194366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5236.195188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.197308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5236.198050] LustreError: Skipped 1 previous similar message [ 5236.203461] Lustre: Skipped 1 previous similar message [ 5241.314188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5241.317296] Lustre: Skipped 2 previous similar messages [ 5242.258208] Lustre: server umount lustre-OST0000 complete [ 5246.433478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.434037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.438749] Lustre: Skipped 1 previous similar message [ 5249.625039] Lustre: server umount lustre-MDT0000 complete [ 5250.821259] LustreError: 212726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439267 with bad export cookie 17017032093002516865 [ 5250.824042] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.824410] LustreError: 212726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5250.942989] Lustre: server umount lustre-MDT0001 complete [ 5254.362453] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5255.307695] Key type lgssc unregistered [ 5255.428407] LNet: 214950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5255.430783] LNetError: 214950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5255.441687] LNet: Removed LNI 192.168.201.148@tcp [ 5255.719285] Key type .llcrypt unregistered [ 5255.721114] Key type ._llcrypt unregistered [ 5264.329497] Key type ._llcrypt registered [ 5264.330382] Key type .llcrypt registered [ 5264.363912] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5264.656500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5264.681077] alg: No test for adler32 (adler32-zlib) [ 5265.526466] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5265.601776] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5267.176096] Key type lgssc registered [ 5267.494610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5269.858684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5269.863833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.013690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.181405] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5275.044834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.189145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5276.506326] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5278.330180] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5279.643602] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5281.968344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5282.103963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.060393] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5285.955721] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5296.423626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5305.825782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5305.828912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.833736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5306.849732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5306.851643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5306.852189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5309.262644] Lustre: server umount lustre-OST0000 complete [ 5310.944854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5310.948617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.953369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5310.954801] Lustre: Skipped 1 previous similar message [ 5316.637800] Lustre: server umount lustre-MDT0000 complete [ 5317.089292] LustreError: 216322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5317.093726] LustreError: 216322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5317.753992] LustreError: 216303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439334 with bad export cookie 17665662505501704791 [ 5317.757300] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5317.758274] LustreError: 216303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5317.860175] Lustre: server umount lustre-MDT0001 complete [ 5321.112197] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5322.192396] Key type lgssc unregistered [ 5322.323346] LNet: 219673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5322.326548] LNetError: 219673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5322.337255] LNet: Removed LNI 192.168.201.148@tcp [ 5322.640473] Key type .llcrypt unregistered [ 5322.641625] Key type ._llcrypt unregistered [ 5330.872195] Key type ._llcrypt registered [ 5330.873677] Key type .llcrypt registered [ 5330.908423] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5331.314532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5331.325265] alg: No test for adler32 (adler32-zlib) [ 5332.175695] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5332.261524] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5333.840136] Key type lgssc registered [ 5334.211774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5600.449318] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5601.377464] Key type lgssc unregistered [ 5601.491330] LNet: 221638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5601.493314] LNetError: 221638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5601.506289] LNet: Removed LNI 192.168.201.148@tcp [ 5601.749446] Key type .llcrypt unregistered [ 5601.750380] Key type ._llcrypt unregistered [ 5608.928448] Key type ._llcrypt registered [ 5608.929377] Key type .llcrypt registered [ 5608.961610] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5614.858426] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5615.156269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.182061] alg: No test for adler32 (adler32-zlib) [ 5616.028609] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5616.106074] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5617.680092] Key type lgssc registered [ 5617.990506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5626.807314] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5271 sec ========== 03:20:43 (1772439643) [ 5627.315877] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:20:43 (1772439643) === [ 5628.270744] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:20:44 (1772439644) === [ 5640.331209] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5641.217433] Key type lgssc unregistered [ 5641.330288] LNet: 226791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5641.332527] LNetError: 226791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5641.339238] LNet: Removed LNI 192.168.201.148@tcp [ 5641.576397] Key type .llcrypt unregistered [ 5641.577431] Key type ._llcrypt unregistered