[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 383591562 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002315] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007012] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008004] pid_max: default: 32768 minimum: 301 [ 0.009108] LSM: Security Framework initializing [ 0.010039] Yama: becoming mindful. [ 0.011023] SELinux: Initializing. [ 0.012004] *** VALIDATE selinux *** [ 0.019100] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023526] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024103] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026079] *** VALIDATE tmpfs *** [ 0.027478] *** VALIDATE proc *** [ 0.029082] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.031219] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032129] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034026] Spectre V2 : User space: Vulnerable [ 0.035004] Speculative Store Bypass: Vulnerable [ 0.037735] debug: unmapping init [mem 0xffffffff96659000-0xffffffff96660fff] [ 0.040000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040613] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041026] ... version: 2 [ 0.042009] ... bit width: 48 [ 0.043008] ... generic registers: 4 [ 0.044009] ... value mask: 0000ffffffffffff [ 0.045009] ... max period: 00007fffffffffff [ 0.046010] ... fixed-purpose events: 3 [ 0.047007] ... event mask: 000000070000000f [ 0.048263] rcu: Hierarchical SRCU implementation. [ 0.050442] smp: Bringing up secondary CPUs ... [ 0.051556] x86: Booting SMP configuration: [ 0.052022] .... node #0, CPUs: #1 #2 #3 [ 0.055078] smp: Brought up 1 node, 4 CPUs [ 0.057007] smpboot: Max logical packages: 1 [ 0.058010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.248696] node 0 deferred pages initialised in 189ms [ 0.252134] devtmpfs: initialized [ 0.253236] x86/mm: Memory block size: 128MB [ 0.255513] gcov: version magic: 0x41383552 [ 0.257117] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.258048] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.259213] pinctrl core: initialized pinctrl subsystem [ 0.260123] [ 0.260448] ************************************************************* [ 0.261005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262005] ** ** [ 0.263007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264009] ** ** [ 0.265008] ** This means that this kernel is built to expose internal ** [ 0.266008] ** IOMMU data structures, which may compromise security on ** [ 0.267009] ** your system. ** [ 0.268010] ** ** [ 0.269006] ** If you see this message and you are not debugging the ** [ 0.270005] ** kernel, report this immediately to your vendor! ** [ 0.271005] ** ** [ 0.272008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273007] ************************************************************* [ 0.274466] NET: Registered protocol family 16 [ 0.275283] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.278322] cpuidle: using governor menu [ 0.279167] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281303] PCI: Using configuration type 1 for base access [ 0.283086] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.289243] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.291016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293106] cryptd: max_cpu_qlen set to 1000 [ 0.295235] ACPI: Added _OSI(Module Device) [ 0.296017] ACPI: Added _OSI(Processor Device) [ 0.297008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297952] ACPI: Added _OSI(Processor Aggregator Device) [ 0.301062] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305334] ACPI: Interpreter enabled [ 0.306030] ACPI: PM: (supports S0 S3 S4 S5) [ 0.307008] ACPI: Using IOAPIC for interrupt routing [ 0.309048] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311326] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.318266] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.320019] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.322007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.324037] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.328255] acpiphp: Slot [2] registered [ 0.329071] acpiphp: Slot [5] registered [ 0.329938] acpiphp: Slot [6] registered [ 0.332081] acpiphp: Slot [7] registered [ 0.333104] acpiphp: Slot [8] registered [ 0.334049] acpiphp: Slot [9] registered [ 0.334861] acpiphp: Slot [10] registered [ 0.336060] acpiphp: Slot [3] registered [ 0.336851] acpiphp: Slot [4] registered [ 0.337039] acpiphp: Slot [11] registered [ 0.337862] acpiphp: Slot [12] registered [ 0.339048] acpiphp: Slot [13] registered [ 0.339786] acpiphp: Slot [14] registered [ 0.340039] acpiphp: Slot [15] registered [ 0.340807] acpiphp: Slot [16] registered [ 0.342040] acpiphp: Slot [17] registered [ 0.343020] acpiphp: Slot [18] registered [ 0.343858] acpiphp: Slot [19] registered [ 0.344039] acpiphp: Slot [20] registered [ 0.344827] acpiphp: Slot [21] registered [ 0.346039] acpiphp: Slot [22] registered [ 0.346800] acpiphp: Slot [23] registered [ 0.347038] acpiphp: Slot [24] registered [ 0.347823] acpiphp: Slot [25] registered [ 0.349041] acpiphp: Slot [26] registered [ 0.349751] acpiphp: Slot [27] registered [ 0.350042] acpiphp: Slot [28] registered [ 0.350805] acpiphp: Slot [29] registered [ 0.352039] acpiphp: Slot [30] registered [ 0.352798] acpiphp: Slot [31] registered [ 0.353031] PCI host bridge to bus 0000:00 [ 0.353776] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.359009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.360009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361068] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362599] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364689] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369917] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372477] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378582] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381696] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385705] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.390010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406126] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410816] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414833] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426739] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.435136] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.440012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.462000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.470015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.477012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.496286] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.500849] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.518014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.525249] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.529008] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.533009] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.543985] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.550078] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.555874] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.558007] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.569835] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.577714] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.580258] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.581256] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.583311] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.586180] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.591026] iommu: Default domain type: Passthrough [ 0.592515] SCSI subsystem initialized [ 0.593063] ACPI: bus type USB registered [ 0.593925] usbcore: registered new interface driver usbfs [ 0.595028] usbcore: registered new interface driver hub [ 0.596034] usbcore: registered new device driver usb [ 0.597098] pps_core: LinuxPPS API ver. 1 registered [ 0.598007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.601063] PTP clock support registered [ 0.602141] EDAC MC: Ver: 3.0.0 [ 0.603343] PCI: Using ACPI for IRQ routing [ 0.604670] NetLabel: Initializing [ 0.606009] NetLabel: domain hash size = 128 [ 0.608006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.610076] NetLabel: unlabeled traffic allowed by default [ 0.612114] vgaarb: loaded [ 0.613216] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.614007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.618003] clocksource: Switched to clocksource kvm-clock [ 0.701555] VFS: Disk quotas dquot_6.6.0 [ 0.702945] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.704490] *** VALIDATE ramfs *** [ 0.705357] *** VALIDATE hugetlbfs *** [ 0.706330] pnp: PnP ACPI init [ 0.708021] pnp: PnP ACPI: found 6 devices [ 0.721417] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723888] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.725445] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.726983] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.728375] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.730119] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.732499] NET: Registered protocol family 2 [ 0.734690] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.739084] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.742732] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.746460] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.749709] TCP: Hash tables configured (established 65536 bind 65536) [ 0.751578] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.753328] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754911] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.756574] NET: Registered protocol family 1 [ 0.758116] RPC: Registered named UNIX socket transport module. [ 0.759282] RPC: Registered udp transport module. [ 0.760257] RPC: Registered tcp transport module. [ 0.761129] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.762288] NET: Registered protocol family 44 [ 0.763318] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.764566] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.765685] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.766964] PCI: CLS 0 bytes, default 64 [ 0.768060] Unpacking initramfs... [ 2.155258] debug: unmapping init [mem 0xffff9c24bcc54000-0xffff9c24bffbffff] [ 2.160625] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.162572] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.164962] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.647582] Initialise system trusted keyrings [ 2.648659] Key type blacklist registered [ 2.649850] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.657086] zbud: loaded [ 2.661701] *** VALIDATE nfs *** [ 2.663135] *** VALIDATE nfs4 *** [ 2.664994] pstore: using deflate compression [ 2.669157] Platform Keyring initialized [ 2.767618] NET: Registered protocol family 38 [ 2.769411] Key type asymmetric registered [ 2.770718] Asymmetric key parser 'x509' registered [ 2.772548] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.777768] io scheduler mq-deadline registered [ 2.779163] io scheduler kyber registered [ 2.780499] io scheduler bfq registered [ 2.781947] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.784297] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.787179] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.789569] ACPI: Power Button [PWRF] [ 2.794804] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.801295] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.815047] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.821638] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.835894] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.868028] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.901059] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.906130] Non-volatile memory driver v1.3 [ 2.907917] Linux agpgart interface v0.103 [ 2.933519] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.936575] vda: detected capacity change from 0 to 74698752 [ 2.950329] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.953399] vdb: detected capacity change from 0 to 1073741824 [ 2.966342] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.969166] vdc: detected capacity change from 0 to 2621440000 [ 2.981409] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.984094] vdd: detected capacity change from 0 to 2621440000 [ 2.996771] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.998889] vde: detected capacity change from 0 to 4294967296 [ 3.015684] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.021444] vdf: detected capacity change from 0 to 4294967296 [ 3.029642] libphy: Fixed MDIO Bus: probed [ 3.035619] usbcore: registered new interface driver usbserial_generic [ 3.038308] usbserial: USB Serial support registered for generic [ 3.040733] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.045064] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.047243] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.049859] mousedev: PS/2 mouse device common for all mice [ 3.052899] rtc_cmos 00:05: RTC can wake from S4 [ 3.056343] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.056855] rtc_cmos 00:05: registered as rtc0 [ 3.061869] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.063064] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.063116] intel_pstate: CPU model not supported [ 3.067171] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.073250] hid: raw HID events driver (C) Jiri Kosina [ 3.074991] usbcore: registered new interface driver usbhid [ 3.076674] usbhid: USB HID core driver [ 3.077633] drop_monitor: Initializing network drop monitor service [ 3.079097] Initializing XFRM netlink socket [ 3.081271] NET: Registered protocol family 10 [ 3.083897] Segment Routing with IPv6 [ 3.085257] NET: Registered protocol family 17 [ 3.087278] mpls_gso: MPLS GSO support [ 3.093276] RAS: Correctable Errors collector initialized. [ 3.095644] AVX version of gcm_enc/dec engaged. [ 3.097050] AES CTR mode by8 optimization enabled [ 3.155292] sched_clock: Marking stable (3155274152, 0)->(3892763974, -737489822) [ 3.158234] registered taskstats version 1 [ 3.159940] Loading compiled-in X.509 certificates [ 3.164323] zswap: loaded using pool lzo/zbud [ 3.191669] Key type big_key registered [ 3.207177] Key type encrypted registered [ 3.208854] ima: No TPM chip found, activating TPM-bypass! [ 3.210934] ima: Allocated hash algorithm: sha1 [ 3.212592] ima: No architecture policies found [ 3.214464] evm: Initialising EVM extended attributes: [ 3.216178] evm: security.selinux [ 3.217430] evm: security.ima [ 3.218566] evm: security.capability [ 3.219754] evm: HMAC attrs: 0x1 [ 3.222970] rtc_cmos 00:05: setting system clock to 2026-09-14 01:56:50 UTC (1789351010) [ 3.233413] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.237967] debug: unmapping init [mem 0xffffffff96382000-0xffffffff96658fff] [ 3.251142] Write protecting the kernel read-only data: 28672k [ 3.254708] debug: unmapping init [mem 0xffffffff94a03000-0xffffffff94bfffff] [ 3.257407] debug: unmapping init [mem 0xffffffff95314000-0xffffffff953fffff] [ 3.311410] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.320289] systemd[1]: Detected virtualization kvm. [ 3.322041] systemd[1]: Detected architecture x86-64. [ 3.323736] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.350743] systemd[1]: No hostname configured. [ 3.353061] systemd[1]: Set hostname to . [ 3.357968] random: systemd: uninitialized urandom read (16 bytes read) [ 3.360125] systemd[1]: Initializing machine ID from random generator. [ 3.423875] random: ln: uninitialized urandom read (6 bytes read) [ 3.524123] random: systemd: uninitialized urandom read (16 bytes read) [ 3.526455] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.536226] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.539447] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.255792] device-mapper: uevent: version 1.0.3 [ 4.259287] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 5.070422] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.114133] random: fast init done Starting dracut initqueue hook... [ 5.202471] scsi host0: ata_piix [ 5.247488] scsi host1: ata_piix [ 5.249511] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.253020] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.740660] random: crng init done [ 9.743650] random: 7 urandom warning(s) missed due to ratelimiting [ 9.775034] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.530032] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.989289] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.370299] SELinux: Disabled at runtime. [ 15.458174] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.465992] systemd[1]: Detected virtualization kvm. [ 15.467882] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.494329] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.498264] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.505718] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.510796] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.514287] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.524690] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.534341] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ 17.018635] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ 18.785728] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.133791] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.687914] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.731552] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.233914] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.197564] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.743471] NFS: Registering the id_resolver key type [ 26.745759] Key type id_resolver registered [ 26.747613] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 72.933270] libcfs: loading out-of-tree module taints kernel. [ 72.955223] Key type ._llcrypt registered [ 72.956967] Key type .llcrypt registered [ 73.021378] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 82.085149] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 82.747403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.755404] alg: No test for adler32 (adler32-zlib) [ 83.726554] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 84.000322] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 85.607171] Key type lgssc registered [ 86.405942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.203786] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.117150] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 118.915103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 118.926806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 120.037921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 120.058393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 120.111513] Lustre: lustre-MDT0000: new disk, initializing [ 120.147125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 120.159878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 122.181675] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.656454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.702661] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 128.720472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 128.724426] Lustre: Skipped 1 previous similar message [ 128.766693] Lustre: lustre-MDT0001: new disk, initializing [ 128.793128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 128.806141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 128.809437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 130.545637] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.054693] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 137.274729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 137.391189] Lustre: lustre-OST0000: new disk, initializing [ 137.393146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 137.396827] Lustre: 8426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 137.429355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.795511] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 139.897650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 139.903260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 139.948416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 146.877610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 146.947785] Lustre: lustre-OST0001: new disk, initializing [ 146.950657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 146.954341] Lustre: 9497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 146.993136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.737470] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.662282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 156.665054] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 156.689785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 157.394843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 160.331812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 162.664607] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 164.512224] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 166.502986] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 167.655724] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 168.731434] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 169.473270] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 21:59:35 EDT 2026 [ 177.296991] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 177.993342] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 190.431717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 190.436080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.443454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 192.480428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 192.480503] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.483599] Lustre: Skipped 1 previous similar message [ 192.489759] Lustre: Skipped 1 previous similar message [ 196.388983] Lustre: server umount lustre-MDT0000 complete [ 197.882495] LustreError: 6480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351205 with bad export cookie 7030676023788175749 [ 197.883318] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 197.890119] LustreError: 6480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 202.719571] LustreError: 8792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 202.719683] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 202.724521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 202.724538] Lustre: Skipped 2 previous similar messages [ 202.733492] LustreError: 8792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 202.746675] Lustre: Skipped 2 previous similar messages [ 204.092396] Lustre: server umount lustre-MDT0001 complete [ 211.722257] Lustre: server umount lustre-OST0000 complete [ 219.420478] Lustre: server umount lustre-OST0001 complete [ 222.101231] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 225.715302] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 240.114721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.216541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 240.229436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 240.270157] Lustre: lustre-MDT0000: new disk, initializing [ 240.305828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.314905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 241.922672] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 247.063664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.119559] Lustre: 15039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 247.251496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.992660] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 253.618202] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 255.958046] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 258.617653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.748460] Lustre: lustre-OST0000: new disk, initializing [ 258.751095] Lustre: Skipped 1 previous similar message [ 258.753988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.757853] Lustre: Skipped 2 previous similar messages [ 258.760745] Lustre: 17019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.803903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.807601] Lustre: Skipped 1 previous similar message [ 260.183313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 260.190319] Lustre: Skipped 1 previous similar message [ 260.195268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 260.254200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.433804] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 266.407716] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 268.247315] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 268.345062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 270.242704] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 270.332649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 275.936887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 275.942119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.948446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 275.951260] Lustre: Skipped 2 previous similar messages [ 277.228012] Lustre: server umount lustre-OST0000 complete [ 281.057055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.061935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.067581] Lustre: Skipped 2 previous similar messages [ 281.074496] Lustre: Skipped 1 previous similar message [ 291.296500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.300709] Lustre: Skipped 3 previous similar messages [ 293.343145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 293.431862] Lustre: server umount lustre-MDT0000 complete [ 295.413154] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351302 with bad export cookie 7030676023788177282 [ 295.417370] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.421964] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 295.595986] Lustre: server umount lustre-MDT0001 complete [ 297.516562] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:01:43 (1789351303) [ 300.950885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 301.194563] LustreError: 19283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.234204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 303.458869] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 306.657505] LustreError: 19284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.901705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.159885] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 313.665583] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 316.362994] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 319.699556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 322.791465] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 326.612905] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 340.952477] Lustre: server umount lustre-OST0000 complete [ 343.522550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.538184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.541512] Lustre: Skipped 1 previous similar message [ 352.756551] LustreError: 22083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9c253dc2c100) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 352.785472] Lustre: server umount lustre-MDT0000 complete [ 353.760395] LustreError: 19284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.767478] LustreError: 19284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 354.442231] LustreError: 19264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351361 with bad export cookie 7030676023788178164 [ 354.445624] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.450558] LustreError: 19264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 364.091071] Lustre: server umount lustre-MDT0001 complete [ 368.937259] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 370.381927] Key type lgssc unregistered [ 370.542921] LNet: 22839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.547224] LNetError: 22839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 370.559431] LNet: Removed LNI 192.168.203.139@tcp [ 370.998182] Key type .llcrypt unregistered [ 370.999826] Key type ._llcrypt unregistered [ 382.094294] Key type ._llcrypt registered [ 382.095768] Key type .llcrypt registered [ 382.142378] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 382.518088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 382.587209] alg: No test for adler32 (adler32-zlib) [ 383.469842] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 383.580333] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 385.175340] Key type lgssc registered [ 385.646840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 389.159377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 389.169957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.372722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.097801] hrtimer: interrupt took 3793094 ns [ 392.119105] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 395.513410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.670380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 397.227111] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 400.082930] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 402.467321] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 405.348916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.519893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.175925] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 411.312135] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 414.769885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 424.929546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.932837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.935812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.942143] Lustre: Skipped 1 previous similar message [ 424.947873] Lustre: Skipped 1 previous similar message [ 428.498817] Lustre: server umount lustre-OST0000 complete [ 430.049530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.055413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 430.058567] Lustre: Skipped 1 previous similar message [ 435.168485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.173242] Lustre: Skipped 1 previous similar message [ 440.289677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.293858] Lustre: Skipped 1 previous similar message [ 444.383158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.452680] Lustre: server umount lustre-MDT0000 complete [ 445.408096] LustreError: 24955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.419172] LustreError: 24955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 446.334333] LustreError: 27162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351453 with bad export cookie 13948695803423532579 [ 446.334623] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.345153] LustreError: 27162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 446.550267] Lustre: server umount lustre-MDT0001 complete [ 457.160040] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 461.816712] Key type lgssc unregistered [ 462.192339] LNet: 28204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.204056] LNetError: 28204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.275784] LNet: Removed LNI 192.168.203.139@tcp [ 465.512248] Key type .llcrypt unregistered [ 465.520261] Key type ._llcrypt unregistered [ 505.647917] Key type ._llcrypt registered [ 505.651363] Key type .llcrypt registered [ 505.925730] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 507.852353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 508.165522] alg: No test for adler32 (adler32-zlib) [ 510.229923] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 511.022315] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 512.935154] Key type lgssc registered [ 514.433855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 521.877873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 521.903498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.259899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.130261] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 533.473430] LustreError: 29619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.486387] LustreError: 29619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 534.000153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.260575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 537.664486] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 543.487532] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.656166] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.374165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.607537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.858493] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 567.683291] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 567.854634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 572.900902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.910395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 572.911430] Lustre: Skipped 1 previous similar message [ 572.918926] Lustre: Skipped 1 previous similar message [ 574.068356] Lustre: server umount lustre-OST0000 complete [ 578.022456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.027252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.029839] Lustre: Skipped 1 previous similar message [ 578.044363] Lustre: Skipped 1 previous similar message [ 583.140306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.149188] Lustre: Skipped 1 previous similar message [ 588.257118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.264724] Lustre: Skipped 1 previous similar message [ 591.327579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 591.597811] Lustre: server umount lustre-MDT0000 complete [ 593.377621] LustreError: 29615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.392335] LustreError: 29615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 594.463816] LustreError: 29599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351601 with bad export cookie 15193705050158542792 [ 594.473582] LustreError: 29599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 594.477098] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.761231] Lustre: server umount lustre-MDT0001 complete [ 602.314764] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 604.344946] Key type lgssc unregistered [ 604.586611] LNet: 33497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.599882] LNetError: 33497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.615797] LNet: Removed LNI 192.168.203.139@tcp [ 605.247058] Key type .llcrypt unregistered [ 605.250097] Key type ._llcrypt unregistered [ 621.315801] Key type ._llcrypt registered [ 621.318295] Key type .llcrypt registered [ 621.412328] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 622.076351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.149773] alg: No test for adler32 (adler32-zlib) [ 623.141772] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 623.314323] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 624.951151] Key type lgssc registered [ 625.860670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.725926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.738185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.091691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.442172] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 642.834109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.106407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 646.046346] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 650.928863] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 655.324196] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 660.174893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.436100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.593853] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 672.113534] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 678.918380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 691.174177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 691.181426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.198390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.241653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 694.258379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.260595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.280280] Lustre: Skipped 1 previous similar message [ 695.655279] Lustre: server umount lustre-OST0000 complete [ 699.362041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 699.363745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.365460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.385718] Lustre: Skipped 1 previous similar message [ 704.480613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 704.483569] Lustre: Skipped 1 previous similar message [ 709.602210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.608067] Lustre: Skipped 1 previous similar message [ 713.183130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 713.339427] Lustre: server umount lustre-MDT0000 complete [ 714.726600] LustreError: 34882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.743767] LustreError: 34882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 716.431781] LustreError: 34868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351723 with bad export cookie 6442164385503885906 [ 716.434640] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 716.445023] LustreError: 34868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 716.664455] Lustre: server umount lustre-MDT0001 complete [ 724.433550] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 726.331799] Key type lgssc unregistered [ 726.556709] LNet: 38448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.561545] LNetError: 38448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.577192] LNet: Removed LNI 192.168.203.139@tcp [ 727.216479] Key type .llcrypt unregistered [ 727.219111] Key type ._llcrypt unregistered [ 744.403118] Key type ._llcrypt registered [ 744.405711] Key type .llcrypt registered [ 744.491887] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 745.209956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.231316] alg: No test for adler32 (adler32-zlib) [ 746.225151] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 746.446708] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 748.071212] Key type lgssc registered [ 748.983674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 755.276082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 755.296247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.604431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 759.836308] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 765.940826] LustreError: 39840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.961783] LustreError: 39840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 766.318226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.645594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 770.213417] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 776.034320] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 780.087699] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 785.275417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 785.528041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.033499] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 796.235529] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 800.874795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 806.372695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.374333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 806.384183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 806.394628] Lustre: Skipped 1 previous similar message [ 808.405181] Lustre: lustre-OST0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 811.496963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.501902] Lustre: Skipped 1 previous similar message [ 812.430120] Lustre: server umount lustre-OST0000 complete [ 831.969034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.981319] Lustre: Skipped 1 previous similar message [ 831.984987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.988523] Lustre: Skipped 1 previous similar message [ 836.071043] LustreError: 39840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.085434] LustreError: 39840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 836.803728] Lustre: server umount lustre-MDT0000 complete [ 839.813148] LustreError: 39819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351847 with bad export cookie 4375653426648368825 [ 839.817420] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.827634] LustreError: 39819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 846.061221] Lustre: server umount lustre-MDT0001 complete [ 852.405369] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 853.937150] Key type lgssc unregistered [ 854.091829] LNet: 43549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.099088] LNetError: 43549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.106422] LNet: Removed LNI 192.168.203.139@tcp [ 854.554142] Key type .llcrypt unregistered [ 854.557560] Key type ._llcrypt unregistered [ 868.928552] Key type ._llcrypt registered [ 868.933700] Key type .llcrypt registered [ 869.010708] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 869.658157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 869.723738] alg: No test for adler32 (adler32-zlib) [ 870.667643] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 870.822768] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 872.431259] Key type lgssc registered [ 873.073505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 878.435314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 878.461488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.740958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.511210] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 888.491917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.734753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 891.766151] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 896.773656] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 900.837855] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.053962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.302980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.209634] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 916.665386] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 925.740525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 933.824969] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 934.757707] Lustre: server umount lustre-MDT0000 complete [ 934.882306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 934.889798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.907162] LustreError: 44938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.931770] LustreError: 44938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 935.913157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.917483] LustreError: 44957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.939724] LustreError: 44957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 937.510311] LustreError: 45672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789351944 with bad export cookie 10582934942388040749 [ 937.513019] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.520359] LustreError: 45672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 938.924307] LustreError: 46781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.947912] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 939.772513] Lustre: server umount lustre-MDT0001 complete [ 951.483259] LustreError: 44921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.39@tcp arrived at 1789351958 with bad export cookie 10582934942388041414 [ 951.497212] LustreError: 44921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 969.655316] Lustre: server umount lustre-OST0000 complete [ 981.401796] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 983.462506] Key type lgssc unregistered [ 983.689548] LNet: 48826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.695812] LNetError: 48826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.709789] LNet: Removed LNI 192.168.203.139@tcp [ 984.369108] Key type .llcrypt unregistered [ 984.371136] Key type ._llcrypt unregistered [ 1003.062302] Key type ._llcrypt registered [ 1003.063889] Key type .llcrypt registered [ 1003.137832] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1003.960319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.981383] alg: No test for adler32 (adler32-zlib) [ 1004.926758] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1005.117539] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1006.751203] Key type lgssc registered [ 1007.555372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1013.208032] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1013.226301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1028.767455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1033.887345] LustreError: 50193:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 1060.575185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.305596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1070.467575] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1236.609298] Lustre: server umount lustre-OST0000 complete [ 1246.502050] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1248.242972] Key type lgssc unregistered [ 1248.458652] LNet: 51867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.466149] LNetError: 51867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.518718] LNet: Removed LNI 192.168.203.139@tcp [ 1250.028184] Key type .llcrypt unregistered [ 1250.030519] Key type ._llcrypt unregistered [ 1264.321789] Key type ._llcrypt registered [ 1264.324425] Key type .llcrypt registered [ 1264.410848] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1265.017416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.066319] alg: No test for adler32 (adler32-zlib) [ 1266.018406] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1266.154656] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1267.767147] Key type lgssc registered [ 1268.514930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1273.734904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1273.748497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.105174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.145637] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1284.112720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.372776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.323321] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1292.262618] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1295.897553] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1300.072425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1300.304670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.377429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1304.681963] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1309.602659] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1311.712384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1311.719705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.730348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1314.785033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1314.787107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1314.789620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.111075] Lustre: server umount lustre-OST0000 complete [ 1319.906283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.920759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.923467] Lustre: Skipped 2 previous similar messages [ 1325.026397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.032660] Lustre: Skipped 1 previous similar message [ 1325.582727] Lustre: server umount lustre-MDT0000 complete [ 1327.759642] LustreError: 53239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789352335 with bad export cookie 627688830889002294 [ 1327.764729] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.771333] LustreError: 53239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1327.961367] Lustre: server umount lustre-MDT0001 complete [ 1334.024183] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1335.806914] Key type lgssc unregistered [ 1336.004890] LNet: 56792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.013793] LNetError: 56792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.029092] LNet: Removed LNI 192.168.203.139@tcp [ 1336.570513] Key type .llcrypt unregistered [ 1336.572363] Key type ._llcrypt unregistered [ 1351.892717] Key type ._llcrypt registered [ 1351.894872] Key type .llcrypt registered [ 1351.974227] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1352.719867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1352.744157] alg: No test for adler32 (adler32-zlib) [ 1353.703938] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1353.851072] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1355.495159] Key type lgssc registered [ 1356.161369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.852520] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.863624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.352085] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1381.471743] LustreError: 58156:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 1408.095277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.559858] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1415.649592] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.478338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.833754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.886567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1422.514262] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1427.523203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.744146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1430.099248] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1434.248214] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.518852] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1440.229059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1440.238328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.256455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.296923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.301742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.302968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.422626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1448.428688] Lustre: Skipped 1 previous similar message [ 1453.023172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1453.123825] Lustre: server umount lustre-OST0000 complete [ 1471.006543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.020326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1471.024310] Lustre: Skipped 2 previous similar messages [ 1478.114475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.117934] Lustre: Skipped 3 previous similar messages [ 1485.280155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1485.374027] Lustre: server umount lustre-MDT0000 complete [ 1487.414527] LustreError: 58164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789352494 with bad export cookie 17599503534817853682 [ 1487.418299] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1487.421497] LustreError: 58164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1493.657794] Lustre: server umount lustre-MDT0001 complete [ 1499.118337] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1500.684435] Key type lgssc unregistered [ 1500.896802] LNet: 61907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.903392] LNetError: 61907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.919458] LNet: Removed LNI 192.168.203.139@tcp [ 1501.424397] Key type .llcrypt unregistered [ 1501.426586] Key type ._llcrypt unregistered [ 1514.928211] Key type ._llcrypt registered [ 1514.929528] Key type .llcrypt registered [ 1514.989526] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1515.643920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.656731] alg: No test for adler32 (adler32-zlib) [ 1516.563971] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1516.701473] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1518.311611] Key type lgssc registered [ 1518.971970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1523.403897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1523.417397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.668883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.291092] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1532.747825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.989337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.591418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1539.887767] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1543.158830] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1546.844232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1547.035140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.101761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1549.938904] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1554.162322] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1568.742873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1568.743574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.745117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1568.755982] Lustre: Skipped 1 previous similar message [ 1573.860644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.867349] Lustre: Skipped 1 previous similar message [ 1574.781914] Lustre: server umount lustre-OST0000 complete [ 1578.977034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1578.980888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1578.982966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.991116] Lustre: Skipped 1 previous similar message [ 1578.998943] Lustre: Skipped 1 previous similar message [ 1584.096432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.105447] Lustre: Skipped 1 previous similar message [ 1589.218790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.224974] Lustre: Skipped 1 previous similar message [ 1591.263138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1591.358403] Lustre: server umount lustre-MDT0000 complete [ 1593.403083] LustreError: 63280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789352600 with bad export cookie 16223674251016410773 [ 1593.409577] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.409778] LustreError: 63280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1593.559991] Lustre: server umount lustre-MDT0001 complete [ 1599.488525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1601.142778] Key type lgssc unregistered [ 1601.334186] LNet: 66853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.340163] LNetError: 66853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.353204] LNet: Removed LNI 192.168.203.139@tcp [ 1601.827524] Key type .llcrypt unregistered [ 1601.831919] Key type ._llcrypt unregistered [ 1616.132863] Key type ._llcrypt registered [ 1616.134197] Key type .llcrypt registered [ 1616.191948] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1616.721236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1616.730363] alg: No test for adler32 (adler32-zlib) [ 1617.639192] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1617.752541] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1619.359163] Key type lgssc registered [ 1619.991040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1624.632409] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:23:51 (1789352631) [ 1628.124232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.136771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.369910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.314255] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1635.765426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.959386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.173379] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1641.955832] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1644.539587] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1647.913919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1648.097434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.245831] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1654.251871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1655.163982] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1663.812635] Lustre: Failing over lustre-MDT0000 [ 1663.988948] Lustre: server umount lustre-MDT0000 complete [ 1666.530380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1666.531814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.533032] LustreError: 70375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.533041] LustreError: 70375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1666.565712] Lustre: Skipped 2 previous similar messages [ 1668.025163] LustreError: 68535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.040971] LustreError: 68535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1668.775044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.846525] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.942265] Lustre: *** cfs_fail_loc=135, val=0*** [ 1669.005362] LustreError: 71621:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1669.009939] LustreError: 71621:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -2 [ 1669.015590] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1669.015590] [ 1669.022924] LustreError: MGC192.168.203.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1669.030638] LustreError: 71610:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1669.035655] LustreError: 71610:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1669.040428] LustreError: 71610:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1669.044669] LustreError: 71610:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1669.063326] Lustre: server umount lustre-MDT0000 complete [ 1669.066272] LustreError: 71610:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1671.649735] LustreError: 70375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.662369] LustreError: 70375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1672.601127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.665390] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.797721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1672.827726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1673.130058] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1674.889345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1677.800179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1677.815803] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1677.842870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1681.889227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.895107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.902084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1682.913545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1682.919597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.925701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.541917] Lustre: server umount lustre-OST0000 complete [ 1688.033616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.033930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1688.034647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.034650] Lustre: Skipped 1 previous similar message [ 1688.046493] Lustre: Skipped 1 previous similar message [ 1693.152855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.157083] Lustre: Skipped 2 previous similar messages [ 1693.408621] Lustre: server umount lustre-MDT0000 complete [ 1695.175880] LustreError: 71113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789352702 with bad export cookie 10926335960795121815 [ 1695.178047] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1695.182317] LustreError: 71113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1695.328792] Lustre: server umount lustre-MDT0001 complete [ 1699.668581] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1700.904037] Key type lgssc unregistered [ 1701.048614] LNet: 73399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.051650] LNetError: 73399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.059278] LNet: Removed LNI 192.168.203.139@tcp [ 1701.454947] Key type .llcrypt unregistered [ 1701.456437] Key type ._llcrypt unregistered [ 1713.146603] Key type ._llcrypt registered [ 1713.150411] Key type .llcrypt registered [ 1713.214621] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1713.668885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.744792] alg: No test for adler32 (adler32-zlib) [ 1714.683384] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1714.825403] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1716.447164] Key type lgssc registered [ 1717.082395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.927886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.942446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.178081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.073159] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1728.415077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.595512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1730.522914] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1734.708451] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1737.885540] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1741.910826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1742.106597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.405433] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1747.247868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1749.570734] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1758.166402] Lustre: Failing over lustre-MDT0000 [ 1758.346423] Lustre: server umount lustre-MDT0000 complete [ 1758.692544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.696370] LustreError: 75948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1758.698510] Lustre: Skipped 2 previous similar messages [ 1758.708562] LustreError: 75948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1762.229382] LustreError: 75948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.568804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.650355] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.734453] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1763.808034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1763.814020] Lustre: Skipped 2 previous similar messages [ 1763.814351] LustreError: 77871:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1763.826270] LustreError: 77871:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -9 [ 1763.833527] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1763.833527] [ 1763.838693] LustreError: MGC192.168.203.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1763.847441] LustreError: 77860:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1763.851615] LustreError: 77860:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1763.862508] LustreError: 77860:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1763.866401] LustreError: 77860:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1763.882844] Lustre: server umount lustre-MDT0000 complete [ 1763.886099] LustreError: 77860:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1767.336557] LustreError: 77097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1767.571893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.671658] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1767.811167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.846449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1770.168481] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1772.461380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1773.036331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1773.050447] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1773.078741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1778.151850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.154532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1778.168758] Lustre: Skipped 1 previous similar message [ 1778.170414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1778.174642] Lustre: Skipped 1 previous similar message [ 1781.266585] Lustre: server umount lustre-OST0000 complete [ 1788.385813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.387164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1788.387203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.392369] Lustre: Skipped 1 previous similar message [ 1789.523729] Lustre: server umount lustre-MDT0000 complete [ 1791.221463] LustreError: 74769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789352798 with bad export cookie 17763353399644538257 [ 1791.223908] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.225615] LustreError: 74769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1791.372872] Lustre: server umount lustre-MDT0001 complete [ 1796.053558] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1797.343461] Key type lgssc unregistered [ 1797.498900] LNet: 79650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.504727] LNetError: 79650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.515709] LNet: Removed LNI 192.168.203.139@tcp [ 1797.923172] Key type .llcrypt unregistered [ 1797.924936] Key type ._llcrypt unregistered [ 1804.350804] Key type ._llcrypt registered [ 1804.352011] Key type .llcrypt registered [ 1804.418551] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1804.598721] Key type .llcrypt unregistered [ 1804.600446] Key type ._llcrypt unregistered [ 1827.264056] Key type ._llcrypt registered [ 1827.265824] Key type .llcrypt registered [ 1827.360651] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1828.640027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.686522] alg: No test for adler32 (adler32-zlib) [ 1829.848096] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1830.033046] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1831.735214] Key type lgssc registered [ 1833.225702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1842.818208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1843.157840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.918849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.999530] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1855.455947] LustreError: 81555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.477768] LustreError: 81555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1856.485758] LustreError: 81556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.746031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.344519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1866.357424] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1874.515942] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.002110] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.633507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.133312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.201288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1893.618573] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1899.557591] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1911.778252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.781908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.783425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.792242] Lustre: Skipped 1 previous similar message [ 1911.807619] LustreError: Skipped 1 previous similar message [ 1916.899900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.902724] Lustre: Skipped 2 previous similar messages [ 1917.420074] Lustre: server umount lustre-OST0000 complete [ 1922.019174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.024026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.040035] Lustre: Skipped 1 previous similar message [ 1922.055118] Lustre: Skipped 1 previous similar message [ 1926.950725] Lustre: server umount lustre-MDT0000 complete [ 1927.143629] LustreError: 81551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.155991] LustreError: 81551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1929.940523] LustreError: 81536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789352937 with bad export cookie 8648936022912443716 [ 1929.948467] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.951832] LustreError: 81536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1930.182328] Lustre: server umount lustre-MDT0001 complete [ 1939.136107] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1941.286526] Key type lgssc unregistered [ 1941.540699] LNet: 85122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.549288] LNetError: 85122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.559970] LNet: Removed LNI 192.168.203.139@tcp [ 1942.180469] Key type .llcrypt unregistered [ 1942.183275] Key type ._llcrypt unregistered [ 1959.373336] Key type ._llcrypt registered [ 1959.376236] Key type .llcrypt registered [ 1959.471159] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1960.314899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.334441] alg: No test for adler32 (adler32-zlib) [ 1961.305614] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 1961.504781] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1963.136956] Key type lgssc registered [ 1963.908236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.129828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.149949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.595570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.135245] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1980.900200] LustreError: 86515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.919351] LustreError: 86515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1981.789451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.111822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.697330] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1991.237060] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1995.297746] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2000.657065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.987309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.126939] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2011.568367] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2016.380919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2021.858166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.860216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.867679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.876274] Lustre: Skipped 1 previous similar message [ 2021.885912] Lustre: Skipped 1 previous similar message [ 2022.575822] Lustre: server umount lustre-OST0000 complete [ 2026.976527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.978206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.985502] Lustre: Skipped 1 previous similar message [ 2026.995606] Lustre: Skipped 1 previous similar message [ 2031.990280] Lustre: server umount lustre-MDT0000 complete [ 2032.096189] LustreError: 86514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.110130] LustreError: 86514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2035.090542] LustreError: 86495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353042 with bad export cookie 6529273505993121977 [ 2035.100469] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.103354] LustreError: 86495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2035.432357] Lustre: server umount lustre-MDT0001 complete [ 2044.184303] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2046.699294] Key type lgssc unregistered [ 2047.015350] LNet: 90047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.022976] LNetError: 90047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.034354] LNet: Removed LNI 192.168.203.139@tcp [ 2047.753815] Key type .llcrypt unregistered [ 2047.759101] Key type ._llcrypt unregistered [ 2066.840031] Key type ._llcrypt registered [ 2066.844240] Key type .llcrypt registered [ 2066.920984] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2067.678697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2067.699769] alg: No test for adler32 (adler32-zlib) [ 2068.665352] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 2068.882518] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2070.543421] Key type lgssc registered [ 2071.666401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2078.619416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2078.656191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.112210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.668272] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2090.477391] LustreError: 91439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.488089] LustreError: 91439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2091.054588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.387965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2095.273250] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2101.539790] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2106.757118] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.589669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2112.989880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.636982] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2125.098965] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2125.302844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2135.521202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2135.524976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.531194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2135.546654] Lustre: Skipped 1 previous similar message [ 2135.558737] Lustre: Skipped 1 previous similar message [ 2136.943849] Lustre: server umount lustre-OST0000 complete [ 2140.644801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.652320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.668644] Lustre: Skipped 1 previous similar message [ 2140.680246] Lustre: Skipped 1 previous similar message [ 2145.715491] Lustre: server umount lustre-MDT0000 complete [ 2145.766078] LustreError: 91439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.781406] LustreError: 91439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2148.535667] LustreError: 91418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353155 with bad export cookie 18267036470286915550 [ 2148.541621] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.543237] LustreError: 91418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2148.893259] Lustre: server umount lustre-MDT0001 complete [ 2156.341930] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2158.012489] Key type lgssc unregistered [ 2158.174313] LNet: 94973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.184734] LNetError: 94973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.198664] LNet: Removed LNI 192.168.203.139@tcp [ 2158.708464] Key type .llcrypt unregistered [ 2158.710504] Key type ._llcrypt unregistered [ 2172.466620] Key type ._llcrypt registered [ 2172.468326] Key type .llcrypt registered [ 2172.515859] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2172.938306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.946128] alg: No test for adler32 (adler32-zlib) [ 2173.843192] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 2173.961633] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2175.567225] Key type lgssc registered [ 2176.158779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2180.085496] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2180.099596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2195.551513] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2200.672907] LustreError: 96336:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 2227.295300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.029455] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2233.612407] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2236.298876] Lustre: server umount lustre-OST0000 complete [ 2240.473301] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:34:07 (1789353247) [ 2243.556746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.780538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.722494] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2249.184773] LustreError: 97974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.190230] LustreError: 97974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2249.711223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.886395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.733671] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2255.123952] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2257.769789] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.270083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.467090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.613660] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2268.140513] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2269.672632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2285.028503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2285.028619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.032446] LustreError: Skipped 1 previous similar message [ 2285.033204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.041316] Lustre: Skipped 1 previous similar message [ 2289.649568] Lustre: server umount lustre-OST0000 complete [ 2295.266271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.269240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.280549] Lustre: Skipped 1 previous similar message [ 2295.284182] Lustre: Skipped 1 previous similar message [ 2297.863472] Lustre: server umount lustre-MDT0000 complete [ 2299.874764] LustreError: 97956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353307 with bad export cookie 2601364791843888816 [ 2299.878082] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.883945] LustreError: 97956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2300.083430] Lustre: server umount lustre-MDT0001 complete [ 2306.024837] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2307.549113] Key type lgssc unregistered [ 2307.724984] LNet: 101685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.732396] LNetError: 101685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2307.746479] LNet: Removed LNI 192.168.203.139@tcp [ 2308.187083] Key type .llcrypt unregistered [ 2308.188809] Key type ._llcrypt unregistered [ 2332.970606] Key type ._llcrypt registered [ 2332.973168] Key type .llcrypt registered [ 2333.040981] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2333.470009] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.518341] alg: No test for adler32 (adler32-zlib) [ 2334.456685] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 2334.590690] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2336.231189] Key type lgssc registered [ 2336.855629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2340.686120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2340.697663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.943365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.814337] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2348.043467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.243521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2350.282279] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2353.640133] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2355.991813] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2358.872565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2359.040769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.643872] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2364.875381] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2367.272689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2378.722607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.725519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2378.730403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2380.583125] Lustre: server umount lustre-OST0000 complete [ 2383.841853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.842745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.847310] Lustre: Skipped 2 previous similar messages [ 2383.852078] Lustre: Skipped 2 previous similar messages [ 2388.450481] Lustre: server umount lustre-MDT0000 complete [ 2388.960577] LustreError: 104573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.966179] LustreError: 104573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2390.016468] LustreError: 103827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353397 with bad export cookie 7296667327455696213 [ 2390.020380] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.021949] LustreError: 103827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.159622] Lustre: server umount lustre-MDT0001 complete [ 2394.632375] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2395.959967] Key type lgssc unregistered [ 2396.141369] LNet: 107397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.147241] LNetError: 107397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.156482] LNet: Removed LNI 192.168.203.139@tcp [ 2396.535944] Key type .llcrypt unregistered [ 2396.537577] Key type ._llcrypt unregistered [ 2406.169686] Key type ._llcrypt registered [ 2406.171423] Key type .llcrypt registered [ 2406.225561] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2406.662382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.715199] alg: No test for adler32 (adler32-zlib) [ 2407.619714] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 2407.763367] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2409.367169] Key type lgssc registered [ 2409.979825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.839542] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.851081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.344072] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2434.463877] LustreError: 108618:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 2461.087743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.591699] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2466.488348] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.293917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.462989] LustreError: MGC192.168.203.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2469.468956] LustreError: 109641:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2469.473188] LustreError: 109641:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2469.477505] LustreError: 109641:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2469.523723] Lustre: server umount lustre-MDT0000 complete [ 2469.526054] LustreError: 109641:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2483.216578] Lustre: server umount lustre-OST0000 complete [ 2486.901230] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2490.769665] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2505.434280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.530526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2505.545488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2505.584210] Lustre: lustre-MDT0000: new disk, initializing [ 2505.615525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.627411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2507.372448] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2512.445028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.495315] Lustre: 112425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2512.532470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2512.535697] Lustre: Skipped 1 previous similar message [ 2512.576620] Lustre: lustre-MDT0001: new disk, initializing [ 2512.625972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.653058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2512.658590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2514.432042] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2518.828063] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2521.228891] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.071775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.193473] Lustre: lustre-OST0000: new disk, initializing [ 2524.198151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2524.202132] Lustre: 114391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2524.240573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.763784] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2529.817051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2529.823581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2529.884831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2531.452546] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2533.272452] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2533.373338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2535.183755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2535.278102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2540.007980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2540.015479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.026063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.028865] Lustre: Skipped 1 previous similar message [ 2542.313405] Lustre: server umount lustre-OST0000 complete [ 2545.121621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.123663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.131431] Lustre: Skipped 2 previous similar messages [ 2545.137282] Lustre: Skipped 1 previous similar message [ 2550.254433] Lustre: server umount lustre-MDT0000 complete [ 2551.867179] LustreError: 112420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353559 with bad export cookie 8003519414480111227 [ 2551.871885] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2551.879859] LustreError: 112420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2552.041979] Lustre: server umount lustre-MDT0001 complete [ 2555.620594] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:39:22 (1789353562) [ 2557.094402] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2563.566663] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2567.615663] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2583.066852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.162520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2583.176463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2583.217569] Lustre: lustre-MDT0000: new disk, initializing [ 2583.251409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.259386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2584.901540] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2589.883528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.938113] Lustre: 119283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2589.970256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2589.972514] Lustre: Skipped 1 previous similar message [ 2590.008752] Lustre: lustre-MDT0001: new disk, initializing [ 2590.053914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2590.058646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2591.835273] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2596.696719] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2599.087303] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2602.051688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2602.168506] Lustre: lustre-OST0000: new disk, initializing [ 2602.170791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2602.173966] Lustre: 121250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2602.208263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2602.211133] Lustre: Skipped 1 previous similar message [ 2604.626847] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2608.843409] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2610.519896] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2610.683748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2610.687783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2610.726488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2611.638852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2613.206475] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2613.299149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2615.780674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.781798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.782746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.785762] Lustre: Skipped 1 previous similar message [ 2615.790632] LustreError: Skipped 2 previous similar messages [ 2620.149355] Lustre: server umount lustre-OST0000 complete [ 2626.017655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2626.017936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.018580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.018586] Lustre: Skipped 1 previous similar message [ 2626.031541] Lustre: Skipped 1 previous similar message [ 2631.136734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.139248] Lustre: Skipped 2 previous similar messages [ 2636.255139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2636.320640] Lustre: server umount lustre-MDT0000 complete [ 2637.670313] LustreError: 121249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353644 with bad export cookie 8003519414480112095 [ 2637.674359] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2637.675743] LustreError: 121249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2637.808483] Lustre: server umount lustre-MDT0001 complete [ 2641.088240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.287535] LustreError: 123470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.296988] LustreError: 123470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2641.321763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.853403] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2646.131224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.981916] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2651.174368] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2653.531747] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2656.225230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.585794] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2661.487168] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2669.377430] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2672.098146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2672.100984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.103701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.106905] Lustre: Skipped 1 previous similar message [ 2672.112482] Lustre: Skipped 3 previous similar messages [ 2672.541996] Lustre: server umount lustre-OST0000 complete [ 2680.231146] Lustre: server umount lustre-MDT0000 complete [ 2681.887434] LustreError: 123450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353689 with bad export cookie 8003519414480112963 [ 2681.890177] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2681.895132] LustreError: 123450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2682.031833] Lustre: server umount lustre-MDT0001 complete [ 2686.182343] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2687.338372] Key type lgssc unregistered [ 2687.480534] LNet: 127100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.483582] LNetError: 127100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.490413] LNet: Removed LNI 192.168.203.139@tcp [ 2687.854990] Key type .llcrypt unregistered [ 2687.856378] Key type ._llcrypt unregistered [ 2694.738937] Key type ._llcrypt registered [ 2694.740452] Key type .llcrypt registered [ 2694.787525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2702.726738] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2703.073598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.081715] alg: No test for adler32 (adler32-zlib) [ 2703.951287] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 2704.062682] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2705.655118] Key type lgssc registered [ 2706.121078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2721.182949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2721.197975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.303028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2722.320284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.361748] Lustre: lustre-MDT0000: new disk, initializing [ 2722.407520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.415913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2724.361640] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2729.392717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.466151] Lustre: 130228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2729.499361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2729.501829] Lustre: Skipped 1 previous similar message [ 2729.539878] Lustre: lustre-MDT0001: new disk, initializing [ 2729.580424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.598170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2729.602668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2731.374617] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2735.972053] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2738.484731] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2741.666680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.804332] Lustre: lustre-OST0000: new disk, initializing [ 2741.806732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2741.810435] Lustre: 132195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2741.848597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.221241] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2745.605029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2745.609296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2745.647904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2748.517147] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2750.197120] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2750.285140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2751.899626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2751.992710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2756.066235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.069387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2756.070429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.072568] Lustre: Skipped 1 previous similar message [ 2756.076172] LustreError: Skipped 1 previous similar message [ 2761.188515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.191203] Lustre: Skipped 2 previous similar messages [ 2766.308573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2766.313819] Lustre: Skipped 1 previous similar message [ 2767.327762] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2767.400780] Lustre: server umount lustre-OST0000 complete [ 2770.912190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2770.916823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.923991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.545065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.548138] Lustre: Skipped 3 previous similar messages [ 2783.199138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2783.283804] Lustre: server umount lustre-MDT0000 complete [ 2784.864167] LustreError: 130221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353792 with bad export cookie 8019028730998116095 [ 2784.868261] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.870791] LustreError: 130221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2784.998437] Lustre: server umount lustre-MDT0001 complete [ 2788.546487] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:43:15 (1789353795) [ 2791.287890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.552569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.316975] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2796.966978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.025610] LustreError: 134610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.037949] LustreError: 134610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2797.137364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2798.897429] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2802.271834] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2804.709808] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2806.755709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.756627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2806.761516] Lustre: Skipped 2 previous similar messages [ 2806.765058] Lustre: Skipped 2 previous similar messages [ 2811.763747] Lustre: server umount lustre-MDT0000 complete [ 2812.897875] LustreError: 134609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.903110] LustreError: 134609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2813.456503] LustreError: 134591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353820 with bad export cookie 8019028730998116977 [ 2813.461607] LustreError: 134591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2813.461703] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2813.606353] Lustre: server umount lustre-MDT0001 complete [ 2817.014438] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:43:43 (1789353823) [ 2820.072813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2835.551593] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2840.671546] LustreError: 137020:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 2866.271331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.396844] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2871.233406] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2872.073080] Lustre: server umount lustre-OST0000 complete [ 2875.183297] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:44:41 (1789353881) [ 2877.782456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.971813] LustreError: 138462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.538489] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2882.719971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.855537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2882.858823] Lustre: Skipped 1 previous similar message [ 2884.465546] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2887.327336] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2889.334116] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2891.806835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.003762] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2896.636183] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2913.763209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.763788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.765825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.765832] Lustre: Skipped 3 previous similar messages [ 2913.774636] Lustre: Skipped 1 previous similar message [ 2914.846923] Lustre: server umount lustre-MDT0000 complete [ 2916.238501] LustreError: 138488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353923 with bad export cookie 8019028730998117628 [ 2916.242903] LustreError: 138488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2916.243244] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.356101] Lustre: server umount lustre-MDT0001 complete [ 2926.943647] Lustre: server umount lustre-OST0000 complete [ 2930.049865] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:45:36 (1789353936) [ 2932.460987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.639594] LustreError: 142130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.673189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.676443] Lustre: Skipped 1 previous similar message [ 2934.153612] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2937.311052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.934086] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2941.494771] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2943.496062] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2945.910665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2948.162128] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2950.947565] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2952.472523] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2957.290092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2957.663773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2959.267453] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2959.362628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2962.402608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2962.403929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.405289] LustreError: Skipped 1 previous similar message [ 2962.405935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.410470] Lustre: Skipped 2 previous similar messages [ 2962.416863] Lustre: Skipped 3 previous similar messages [ 2966.167690] Lustre: server umount lustre-OST0000 complete [ 2968.031974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2972.641382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.646732] Lustre: Skipped 1 previous similar message [ 2973.718233] Lustre: server umount lustre-MDT0000 complete [ 2975.116178] LustreError: 142112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789353982 with bad export cookie 8019028730998118685 [ 2975.119211] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.120903] LustreError: 142112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2975.247373] Lustre: server umount lustre-MDT0001 complete [ 2978.429810] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:46:25 (1789353985) [ 2981.184203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.639866] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3001.760115] LustreError: 146318:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 3027.359352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.362463] Lustre: Skipped 2 previous similar messages [ 3029.506484] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3032.119604] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3034.636990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.802485] LustreError: 146343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.859555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3036.413501] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3039.687453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.333350] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3043.989437] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3046.038710] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3047.697326] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3047.783374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3049.387167] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3049.468150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3052.513584] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.514703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3052.519069] Lustre: Skipped 1 previous similar message [ 3052.521572] Lustre: Skipped 3 previous similar messages [ 3056.340171] Lustre: server umount lustre-MDT0000 complete [ 3057.632413] LustreError: 147355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.639798] LustreError: 147355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3057.707269] LustreError: 146326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354064 with bad export cookie 8019028730998119693 [ 3057.709906] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3057.712490] LustreError: 146326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3057.829931] Lustre: server umount lustre-MDT0001 complete [ 3069.466442] Lustre: server umount lustre-OST0000 complete [ 3072.399414] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:47:59 (1789354079) [ 3074.747076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.207484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3095.327380] LustreError: 150506:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 3122.842850] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3125.166467] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3127.295607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.426620] LustreError: 150531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.470905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3128.696174] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3131.537029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.086386] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3135.769093] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3137.742259] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3140.234094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3140.306495] Lustre: lustre-OST0001: new disk, initializing [ 3140.308275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3140.310820] Lustre: 153374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3142.410192] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3143.992123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3143.995927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3144.032571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3146.580547] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3148.165818] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3148.246762] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3149.784896] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3149.865650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3152.352223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3152.355846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.361058] Lustre: Skipped 1 previous similar message [ 3156.603803] Lustre: server umount lustre-OST0000 complete [ 3159.526208] LustreError: 150520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.533042] LustreError: 150520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3172.319078] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3172.376870] Lustre: server umount lustre-OST0001 complete [ 3185.120116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.122289] Lustre: Skipped 16 previous similar messages [ 3188.191108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3188.266692] Lustre: server umount lustre-MDT0000 complete [ 3189.714760] LustreError: 150513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354196 with bad export cookie 8019028730998120645 [ 3189.717233] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3189.720611] LustreError: 150513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3189.873031] Lustre: server umount lustre-MDT0001 complete [ 3197.721360] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:50:04 (1789354204) [ 3198.272927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3198.912420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3199.524050] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:50:06 (1789354206) [ 3201.882637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.977361] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3201.989945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3202.063296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.065487] Lustre: Skipped 6 previous similar messages [ 3203.382724] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3206.279041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.322711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3206.335037] Lustre: 156933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3207.817782] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3210.359179] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3212.262534] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3214.475475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.540075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3216.545327] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3216.658964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3219.113327] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3220.602514] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3220.687397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3222.115670] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3222.201821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3227.105923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.109444] LustreError: Skipped 3 previous similar messages [ 3227.111497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.118338] Lustre: Skipped 5 previous similar messages [ 3237.343174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3237.397283] Lustre: server umount lustre-OST0000 complete [ 3251.141545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3253.285369] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3255.825843] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3257.421351] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3267.620919] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3267.627105] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3267.627137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3268.720403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 3270.088148] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3270.163410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3271.964395] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f61840be000.ost_server_uuid 50 [ 3272.446045] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f61840be000.ost_server_uuid in FULL state after 0 sec [ 3272.499943] LustreError: 156939:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.39@tcp) returned error from blocking AST (req@0000000031dd0867 x1876273357065088 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ce64fa1d/0x6f495024eacb6a11 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.39@tcp remote: 0x7e0305ff72c0e5 expref: 5 pid: 156938 timeout: 3372 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3272.511195] LustreError: lustre-MDT0000: A client on nid 192.168.203.39@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3272.514356] LustreError: 156928:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.39@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ce64fa1d/0x6f495024eacb6a11 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.39@tcp remote: 0x7e0305ff72c0e5 expref: 6 pid: 156938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3277.794583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3277.797701] LustreError: Skipped 2 previous similar messages [ 3279.563146] Lustre: server umount lustre-OST0000 complete [ 3286.946350] Lustre: server umount lustre-MDT0000 complete [ 3288.032968] LustreError: 156944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.038561] LustreError: 156944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3288.243294] LustreError: 156969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354295 with bad export cookie 8019028730998121828 [ 3288.246848] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.248505] LustreError: 156969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3294.453565] Lustre: server umount lustre-MDT0001 complete [ 3298.222738] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3299.207971] Key type lgssc unregistered [ 3299.330317] LNet: 163116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3299.332477] LNetError: 163116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3299.339257] LNet: Removed LNI 192.168.203.139@tcp [ 3299.644394] Key type .llcrypt unregistered [ 3299.645703] Key type ._llcrypt unregistered [ 3308.243892] Key type ._llcrypt registered [ 3308.244762] Key type .llcrypt registered [ 3308.281126] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3308.604602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3308.654265] alg: No test for adler32 (adler32-zlib) [ 3309.505544] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3309.589311] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3311.167125] Key type lgssc registered [ 3311.524872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.172479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3314.179545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.324891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.754159] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3319.816336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.944201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.378611] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3323.917151] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3325.900951] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3328.316654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.431518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.350190] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3332.847770] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3340.712792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3345.430972] Lustre: Failing over lustre-MDT0000 [ 3345.662238] Lustre: server umount lustre-MDT0000 complete [ 3345.889944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3345.890158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.890965] LustreError: 166706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.890973] LustreError: 166706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3345.908578] Lustre: Skipped 2 previous similar messages [ 3351.009352] LustreError: 164500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.015029] LustreError: 164500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3356.128234] LustreError: 165919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.133688] LustreError: 165919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3356.591357] LustreError: 164487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.39@tcp arrived at 1789354363 with bad export cookie 18290600022622612287 [ 3356.596107] LustreError: 164487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3358.949735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.989320] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3359.074573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.096287] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3360.480799] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3361.252839] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3361.256498] Lustre: lustre-MDT0000: Denying connection for new client 43077cbe-6e60-4c51-8938-39dd29ff9488 (at 192.168.203.39@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 3364.324935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3369.442272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.443630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3369.447726] Lustre: Skipped 1 previous similar message [ 3374.560391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3374.562977] Lustre: Skipped 2 previous similar messages [ 3379.682374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.685047] Lustre: Skipped 1 previous similar message [ 3382.239135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3382.297509] Lustre: server umount lustre-OST0000 complete [ 3383.549218] LustreError: 168254:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3383.552489] Lustre: 167604:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3383.554915] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3383.560954] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3383.565948] LustreError: 167604:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000091cb5ae0 x1876273991290240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3383.573285] LustreError: 167604:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3383.577029] LustreError: 167604:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3383.581307] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 3383.684335] Lustre: server umount lustre-MDT0000 complete [ 3384.800207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.805658] LustreError: 166996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.946138] LustreError: 164487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354392 with bad export cookie 18290600022622612616 [ 3384.950462] LustreError: 164487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3384.951216] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3391.161408] Lustre: server umount lustre-MDT0001 complete [ 3395.038407] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3396.209985] Key type lgssc unregistered [ 3396.346443] LNet: 169009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3396.350094] LNetError: 169009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3396.358667] LNet: Removed LNI 192.168.203.139@tcp [ 3396.708648] Key type .llcrypt unregistered [ 3396.710039] Key type ._llcrypt unregistered [ 3406.296154] Key type ._llcrypt registered [ 3406.297800] Key type .llcrypt registered [ 3406.340986] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3406.795126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3406.809833] alg: No test for adler32 (adler32-zlib) [ 3407.682686] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3407.786452] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3409.375214] Key type lgssc registered [ 3409.782437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3412.933206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3412.940733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.142738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.702817] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3418.958202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.123383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3420.730617] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3423.452844] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3425.470142] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3427.870162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3427.992241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.110548] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3432.828089] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3439.269676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3449.826442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.826589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.827826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3449.830275] Lustre: Skipped 1 previous similar message [ 3449.832767] LustreError: Skipped 1 previous similar message [ 3454.948252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.950093] Lustre: Skipped 1 previous similar message [ 3459.039156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3459.086183] Lustre: server umount lustre-OST0000 complete [ 3465.184467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3465.184670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.185254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.185258] Lustre: Skipped 1 previous similar message [ 3465.194293] Lustre: Skipped 1 previous similar message [ 3466.454428] Lustre: server umount lustre-MDT0000 complete [ 3467.643222] LustreError: 170379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354474 with bad export cookie 8742184235167901874 [ 3467.646910] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.647697] LustreError: 170379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3467.760925] Lustre: server umount lustre-MDT0001 complete [ 3471.107814] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3472.062017] Key type lgssc unregistered [ 3472.184396] LNet: 173951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3472.187555] LNetError: 173951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3472.196521] LNet: Removed LNI 192.168.203.139@tcp [ 3472.497104] Key type .llcrypt unregistered [ 3472.498146] Key type ._llcrypt unregistered [ 3483.768318] Key type ._llcrypt registered [ 3483.769646] Key type .llcrypt registered [ 3483.804134] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3484.107311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3484.141081] alg: No test for adler32 (adler32-zlib) [ 3484.992780] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3485.071991] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3486.647145] Key type lgssc registered [ 3486.988122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.488820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.495544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.652439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.028605] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3494.717978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.835721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3496.118666] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3498.390612] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3500.110058] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3502.263629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.374622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.176567] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3506.509070] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3515.685529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3520.994095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3520.996097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.996703] LustreError: Skipped 1 previous similar message [ 3520.996807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.001160] Lustre: Skipped 1 previous similar message [ 3524.240494] Lustre: server umount lustre-OST0000 complete [ 3525.599986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3525.603303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.609809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3525.612634] Lustre: Skipped 1 previous similar message [ 3531.232345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.234233] Lustre: Skipped 3 previous similar messages [ 3531.668314] Lustre: server umount lustre-MDT0000 complete [ 3532.852929] LustreError: 176361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354540 with bad export cookie 17439432822438890011 [ 3532.855472] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.856053] LustreError: 176361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3532.975704] Lustre: server umount lustre-MDT0001 complete [ 3536.327267] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3537.338188] Key type lgssc unregistered [ 3537.459517] LNet: 179184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3537.462342] LNetError: 179184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3537.469235] LNet: Removed LNI 192.168.203.139@tcp [ 3537.748521] Key type .llcrypt unregistered [ 3537.749481] Key type ._llcrypt unregistered [ 3545.578287] Key type ._llcrypt registered [ 3545.579308] Key type .llcrypt registered [ 3545.618181] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3545.917171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3545.980069] alg: No test for adler32 (adler32-zlib) [ 3546.831674] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3546.912360] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3548.487107] Key type lgssc registered [ 3548.811872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3552.171079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3552.176538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.288248] Lustre: *** cfs_fail_loc=135, val=0*** [ 3553.325337] LustreError: 180682:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3553.327882] LustreError: 180682:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -2 [ 3553.330837] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3553.330837] [ 3553.333606] LustreError: MGC192.168.203.139@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3553.338638] LustreError: 180644:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3553.341980] LustreError: 180644:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3553.344593] LustreError: 180644:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3553.347778] LustreError: 180644:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3553.378091] Lustre: server umount lustre-MDT0000 complete [ 3553.379467] LustreError: 180644:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3555.502543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.634767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.970061] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3559.629639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.749626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3561.080032] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3563.418180] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3565.123661] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3569.889045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.889961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3569.893634] Lustre: Skipped 1 previous similar message [ 3575.264282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3575.266803] Lustre: Skipped 2 previous similar messages [ 3580.383099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3580.384203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.388111] Lustre: Skipped 1 previous similar message [ 3580.437801] Lustre: server umount lustre-MDT0000 complete [ 3581.617161] LustreError: 181035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354588 with bad export cookie 1043174569859222496 [ 3581.619464] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.621128] LustreError: 181035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3581.743350] Lustre: server umount lustre-MDT0001 complete [ 3585.011904] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3586.016388] Key type lgssc unregistered [ 3586.134345] LNet: 183375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3586.136780] LNetError: 183375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3586.145245] LNet: Removed LNI 192.168.203.139@tcp [ 3586.428414] Key type .llcrypt unregistered [ 3586.429676] Key type ._llcrypt unregistered [ 3592.937854] Key type ._llcrypt registered [ 3592.938819] Key type .llcrypt registered [ 3592.974408] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3593.083512] Key type .llcrypt unregistered [ 3593.084575] Key type ._llcrypt unregistered [ 3599.728898] Key type ._llcrypt registered [ 3599.730051] Key type .llcrypt registered [ 3599.761595] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3600.073945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3600.146812] alg: No test for adler32 (adler32-zlib) [ 3600.993941] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3601.077606] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3602.655099] Key type lgssc registered [ 3603.008493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3605.458058] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3605.466678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.895275] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3626.015294] LustreError: 185486:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 3652.639166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.401723] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3656.633953] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3658.851076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.007160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.027070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3660.226101] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3662.860961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.972867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3664.204131] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3666.499123] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3668.205662] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3669.071351] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3674.591796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.593988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.597715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.666042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.666594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.672531] Lustre: Skipped 1 previous similar message [ 3679.883971] Lustre: server umount lustre-OST0000 complete [ 3682.785647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.786261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.788942] Lustre: Skipped 1 previous similar message [ 3682.792236] Lustre: Skipped 1 previous similar message [ 3687.198883] Lustre: server umount lustre-MDT0000 complete [ 3687.904131] LustreError: 186528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.908935] LustreError: 186528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3688.416531] LustreError: 185494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354695 with bad export cookie 1124541263915367713 [ 3688.418959] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3688.420266] LustreError: 185494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3688.543259] Lustre: server umount lustre-MDT0001 complete [ 3692.402258] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3693.496349] Key type lgssc unregistered [ 3693.627338] LNet: 189403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3693.630247] LNetError: 189403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.644320] LNet: Removed LNI 192.168.203.139@tcp [ 3693.977480] Key type .llcrypt unregistered [ 3693.978616] Key type ._llcrypt unregistered [ 3702.861235] Key type ._llcrypt registered [ 3702.862899] Key type .llcrypt registered [ 3702.900279] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3703.223678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.276095] alg: No test for adler32 (adler32-zlib) [ 3704.134566] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3704.229885] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3705.815142] Key type lgssc registered [ 3706.208223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3708.853736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3708.860039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.000839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.432373] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3714.319488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.451474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3715.855752] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3718.319969] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3720.195591] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3722.542122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3722.663114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.699646] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3727.268236] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3734.949865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3739.212900] Lustre: Failing over lustre-MDT0000 [ 3739.310696] Lustre: server umount lustre-MDT0000 complete [ 3740.131292] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.131936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3740.132819] LustreError: 190807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3740.132832] LustreError: 190807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3740.134402] Lustre: Skipped 2 previous similar messages [ 3743.151169] LustreError: 192991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.159291] LustreError: 192991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3745.248851] LustreError: 190808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.254470] LustreError: 190808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3748.263776] LustreError: 190802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.205550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.248833] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.354391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.382897] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3753.383837] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3753.821505] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3755.416829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3757.540663] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3757.546424] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3757.560952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3761.127306] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3765.727756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3765.729927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.734454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3766.753079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3766.755957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.760861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3767.777726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3767.779587] Lustre: Skipped 1 previous similar message [ 3770.381954] Lustre: server umount lustre-OST0000 complete [ 3772.896689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3772.896763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.897299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.905576] Lustre: Skipped 1 previous similar message [ 3777.686925] Lustre: server umount lustre-MDT0000 complete [ 3778.016653] LustreError: 190807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.022373] LustreError: 190807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3778.800958] LustreError: 190789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354786 with bad export cookie 7274746605912078682 [ 3778.804230] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.804951] LustreError: 190789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3778.907954] Lustre: server umount lustre-MDT0001 complete [ 3782.170130] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3783.158712] Key type lgssc unregistered [ 3783.282335] LNet: 195755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3783.285445] LNetError: 195755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3783.293303] LNet: Removed LNI 192.168.203.139@tcp [ 3783.599426] Key type .llcrypt unregistered [ 3783.600879] Key type ._llcrypt unregistered [ 3792.350607] Key type ._llcrypt registered [ 3792.351774] Key type .llcrypt registered [ 3792.394679] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3792.765633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3792.809302] alg: No test for adler32 (adler32-zlib) [ 3793.650121] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3793.734912] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3795.311092] Key type lgssc registered [ 3795.662964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.138040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3798.145212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.276752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.550846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3803.353661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.474365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3804.849387] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3807.309418] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3809.105901] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3811.334465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.448641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.378583] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3815.793404] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3824.739105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3826.683591] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3828.072563] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3838.655456] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3845.600611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.602900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.608392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.611022] Lustre: Skipped 1 previous similar message [ 3850.720750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.773590] Lustre: server umount lustre-OST0000 complete [ 3854.815811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3854.818645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.822756] Lustre: Skipped 1 previous similar message [ 3854.824956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.827196] Lustre: Skipped 1 previous similar message [ 3855.840830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.069554] Lustre: server umount lustre-MDT0000 complete [ 3859.254095] LustreError: 199707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354866 with bad export cookie 13353274552171744864 [ 3859.258042] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3859.258529] LustreError: 199707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3859.371514] Lustre: server umount lustre-MDT0001 complete [ 3862.907508] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3863.884328] Key type lgssc unregistered [ 3864.001378] LNet: 201032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.003757] LNetError: 201032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3864.016302] LNet: Removed LNI 192.168.203.139@tcp [ 3864.300640] Key type .llcrypt unregistered [ 3864.302125] Key type ._llcrypt unregistered [ 3871.528237] Key type ._llcrypt registered [ 3871.529678] Key type .llcrypt registered [ 3871.568240] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3871.916165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3871.974689] alg: No test for adler32 (adler32-zlib) [ 3872.830723] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3872.923295] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3874.511168] Key type lgssc registered [ 3874.869268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3877.443824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3877.450535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.586218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.888231] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3882.701097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.842351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3884.264248] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3886.667414] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3888.417026] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3890.488171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3890.595726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.379549] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3894.687205] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3902.887073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3906.880943] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3916.989638] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3923.625946] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3931.031775] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3943.906110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.907046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3943.907054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3943.909313] Lustre: Skipped 1 previous similar message [ 3947.280100] Lustre: server umount lustre-OST0000 complete [ 3949.024384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.025427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.028031] Lustre: Skipped 1 previous similar message [ 3949.029727] Lustre: Skipped 1 previous similar message [ 3954.144462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3954.147612] Lustre: Skipped 1 previous similar message [ 3954.712756] Lustre: server umount lustre-MDT0000 complete [ 3955.953101] LustreError: 202257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789354963 with bad export cookie 16108456590415522180 [ 3955.955894] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.957826] LustreError: 202257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3956.078537] Lustre: server umount lustre-MDT0001 complete [ 3959.845065] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3960.899503] Key type lgssc unregistered [ 3961.034443] LNet: 207845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3961.038436] LNetError: 207845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3961.050418] LNet: Removed LNI 192.168.203.139@tcp [ 3961.381432] Key type .llcrypt unregistered [ 3961.383216] Key type ._llcrypt unregistered [ 3969.932555] Key type ._llcrypt registered [ 3969.934223] Key type .llcrypt registered [ 3969.972361] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3970.285320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.354141] alg: No test for adler32 (adler32-zlib) [ 3971.201374] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 3971.282785] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3972.863140] Key type lgssc registered [ 3973.193574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3975.598691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3975.606053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.730682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.917242] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3980.486200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.603629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.832495] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3983.993937] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3985.677872] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.719952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.845992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.671558] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3991.891848] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4001.190514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4010.463860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.466470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.470477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.489531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.491925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.492291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.692052] Lustre: server umount lustre-OST0000 complete [ 4016.608759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.609063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4016.609883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4016.609888] Lustre: Skipped 1 previous similar message [ 4016.612757] Lustre: Skipped 1 previous similar message [ 4021.728510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.730245] Lustre: Skipped 2 previous similar messages [ 4026.848481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4026.850736] Lustre: Skipped 1 previous similar message [ 4028.383126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4028.453088] Lustre: server umount lustre-MDT0000 complete [ 4029.739359] LustreError: 209217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789355037 with bad export cookie 16207246172234978348 [ 4029.742482] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.742919] LustreError: 209217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4029.863378] Lustre: server umount lustre-MDT0001 complete [ 4033.770418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4034.794142] Key type lgssc unregistered [ 4034.914347] LNet: 212940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4034.916621] LNetError: 212940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4034.924236] LNet: Removed LNI 192.168.203.139@tcp [ 4035.192805] Key type .llcrypt unregistered [ 4035.193950] Key type ._llcrypt unregistered [ 4042.329412] Key type ._llcrypt registered [ 4042.330240] Key type .llcrypt registered [ 4042.367145] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4042.760927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4042.783052] alg: No test for adler32 (adler32-zlib) [ 4043.620290] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 4043.696599] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4045.271125] Key type lgssc registered [ 4045.642780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4048.380418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4048.387685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.548666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.000963] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4053.805039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.917545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.180225] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4057.555761] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4059.363527] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4061.357482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4061.466863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.197469] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4065.411132] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.766173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4080.096275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.099100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.102853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.194183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.196219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.198450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.199594] Lustre: Skipped 1 previous similar message [ 4085.386805] Lustre: server umount lustre-OST0000 complete [ 4089.313603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.314741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.317331] Lustre: Skipped 1 previous similar message [ 4089.320987] Lustre: Skipped 1 previous similar message [ 4092.823566] Lustre: server umount lustre-MDT0000 complete [ 4094.112520] LustreError: 214164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789355101 with bad export cookie 9902957671222568847 [ 4094.117622] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.118502] LustreError: 214164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4094.250680] Lustre: server umount lustre-MDT0001 complete [ 4097.808986] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4098.794671] Key type lgssc unregistered [ 4098.916415] LNet: 217886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4098.920189] LNetError: 217886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4098.929333] LNet: Removed LNI 192.168.203.139@tcp [ 4099.242724] Key type .llcrypt unregistered [ 4099.243850] Key type ._llcrypt unregistered [ 4106.430867] Key type ._llcrypt registered [ 4106.432099] Key type .llcrypt registered [ 4106.472794] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4106.933230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4106.950712] alg: No test for adler32 (adler32-zlib) [ 4107.805666] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 4107.904498] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4109.487193] Key type lgssc registered [ 4109.890845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4112.516807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4112.523699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.662375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.932132] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4117.699742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.831388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4119.214360] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4121.624032] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4123.443887] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4125.713905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4125.859429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.874995] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4130.393148] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4138.155898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4143.437744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.464346] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4143.548267] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4145.505178] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4147.985985] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4150.562793] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4161.197882] Lustre: Permanently deactivating lustre-OST0001 [ 4161.203356] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4170.743993] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4176.230889] Lustre: Permanently reactivating lustre-OST0001 [ 4176.237677] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4176.239944] Lustre: Skipped 2 previous similar messages [ 4177.974791] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.979337] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4177.982379] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4177.986778] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4178.002201] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4188.127917] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4188.131714] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.138386] Lustre: Skipped 1 previous similar message [ 4188.141149] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4188.639945] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4188.643239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4190.180085] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4190.183213] Lustre: Skipped 1 previous similar message [ 4192.576197] Lustre: server umount lustre-OST0001 complete [ 4194.784831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.785078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.786913] LustreError: Skipped 1 previous similar message [ 4194.792573] Lustre: Skipped 1 previous similar message [ 4194.793132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.797080] Lustre: Skipped 1 previous similar message [ 4195.298410] LustreError: 220949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.304797] LustreError: 220949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4200.418934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.419699] LustreError: 220947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.421306] Lustre: Skipped 3 previous similar messages [ 4200.431295] LustreError: 220947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4205.537118] LustreError: 221317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.542474] LustreError: 221317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4208.095102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4208.143111] Lustre: server umount lustre-OST0000 complete [ 4210.143875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4210.147356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.151161] Lustre: Skipped 1 previous similar message [ 4210.153183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.155299] Lustre: Skipped 2 previous similar messages [ 4215.523241] Lustre: server umount lustre-MDT0000 complete [ 4215.776413] LustreError: 219129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.780499] LustreError: 219129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4216.815671] LustreError: 219156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789355224 with bad export cookie 13413716195375128008 [ 4216.820213] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4216.822119] LustreError: 219156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4216.956989] Lustre: server umount lustre-MDT0001 complete [ 4220.697826] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4221.650210] Key type lgssc unregistered [ 4221.777355] LNet: 224241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4221.780255] LNetError: 224241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4221.787288] LNet: Removed LNI 192.168.203.139@tcp [ 4222.065808] Key type .llcrypt unregistered [ 4222.067603] Key type ._llcrypt unregistered [ 4235.310018] Key type ._llcrypt registered [ 4235.311134] Key type .llcrypt registered [ 4235.349100] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4235.661651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4235.668840] alg: No test for adler32 (adler32-zlib) [ 4236.522056] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 4236.617918] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4238.199125] Key type lgssc registered [ 4238.562056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4241.102787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4241.113064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.193891] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4242.201424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4242.259618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.609473] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4246.455873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.488721] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4246.500504] Lustre: 226423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4246.592434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4247.954617] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4250.326636] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4252.229553] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4254.546831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.623953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4254.714349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.171095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4256.678540] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4259.203586] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4260.766200] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4268.319287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4277.916502] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4284.440907] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4301.596942] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4301.599247] Lustre: Skipped 1 previous similar message [ 4317.735066] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4317.738328] Lustre: Skipped 1 previous similar message [ 4355.511050] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4355.514472] Lustre: Skipped 4 previous similar messages [ 4424.164913] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4424.167665] Lustre: Skipped 7 previous similar messages [ 4434.401253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4434.401331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.403441] LustreError: Skipped 1 previous similar message [ 4434.408233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4434.410921] Lustre: Skipped 1 previous similar message [ 4438.666362] Lustre: server umount lustre-OST0000 complete [ 4441.055804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4441.058067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.061739] Lustre: Skipped 1 previous similar message [ 4441.063683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.640097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.640539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.759902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.762133] Lustre: Skipped 2 previous similar messages [ 4453.856829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.859144] Lustre: Skipped 1 previous similar message [ 4454.367093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4454.429398] Lustre: server umount lustre-MDT0000 complete [ 4455.782507] LustreError: 226414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789355463 with bad export cookie 15525706305616998697 [ 4455.784983] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.788290] LustreError: 226414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.928193] Lustre: server umount lustre-MDT0001 complete [ 4459.302040] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4460.297024] Key type lgssc unregistered [ 4460.416456] LNet: 231212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4460.419255] LNetError: 231212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4460.428278] LNet: Removed LNI 192.168.203.139@tcp [ 4460.703100] Key type .llcrypt unregistered [ 4460.704719] Key type ._llcrypt unregistered [ 4469.210216] Key type ._llcrypt registered [ 4469.211711] Key type .llcrypt registered [ 4469.254285] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4469.575379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4469.590698] alg: No test for adler32 (adler32-zlib) [ 4470.434347] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 4470.531689] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4472.127175] Key type lgssc registered [ 4472.503414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4474.885809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4474.890895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.017458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.212316] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4479.853331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.959750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4481.145583] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4483.397122] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4485.293721] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.545089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.665579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4489.540603] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4492.000761] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4500.968970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4511.202022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4511.202270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.209481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4511.212363] Lustre: Skipped 1 previous similar message [ 4514.258209] Lustre: server umount lustre-OST0000 complete [ 4515.808067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.811747] LustreError: Skipped 1 previous similar message [ 4515.813931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.818901] Lustre: Skipped 1 previous similar message [ 4515.821460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.440118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.441958] Lustre: Skipped 3 previous similar messages [ 4521.685884] Lustre: server umount lustre-MDT0000 complete [ 4522.888120] LustreError: 234440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789355530 with bad export cookie 9169177945527200082 [ 4522.891547] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.892221] LustreError: 234440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4522.997774] Lustre: server umount lustre-MDT0001 complete [ 4526.579569] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4527.608442] Key type lgssc unregistered [ 4527.737345] LNet: 236306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4527.739632] LNetError: 236306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4527.748240] LNet: Removed LNI 192.168.203.139@tcp [ 4528.071712] Key type .llcrypt unregistered [ 4528.072595] Key type ._llcrypt unregistered [ 4536.500658] Key type ._llcrypt registered [ 4536.501772] Key type .llcrypt registered [ 4536.538980] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4536.891570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4536.948766] alg: No test for adler32 (adler32-zlib) [ 4537.813581] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 4537.911838] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4539.487118] Key type lgssc registered [ 4539.865593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4805.631472] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4806.588244] Key type lgssc unregistered [ 4806.719765] LNet: 238307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4806.724384] LNetError: 238307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4806.732282] LNet: Removed LNI 192.168.203.139@tcp [ 4807.004653] Key type .llcrypt unregistered [ 4807.005574] Key type ._llcrypt unregistered [ 4814.728242] Key type ._llcrypt registered [ 4814.729185] Key type .llcrypt registered [ 4814.760307] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4821.313748] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4821.641795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4821.690116] alg: No test for adler32 (adler32-zlib) [ 4822.535955] Lustre: Lustre: Build Version: 2.17.56_50_ge20528f [ 4822.619888] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4824.199141] Key type lgssc registered [ 4824.550416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4833.972086] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4664 sec ========== 23:17:20 (1789355840) [ 4834.489424] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:17:21 (1789355841) === [ 4835.616389] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:17:22 (1789355842) === [ 4847.993674] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4848.903783] Key type lgssc unregistered [ 4849.016249] LNet: 243515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4849.018335] LNetError: 243515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4849.027231] LNet: Removed LNI 192.168.203.139@tcp [ 4849.290243] Key type .llcrypt unregistered [ 4849.291917] Key type ._llcrypt unregistered