[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 565499677 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003109] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009004] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.023268] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028428] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.032458] *** VALIDATE proc *** [ 0.034085] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.037005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044318] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.046257] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047756] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052009] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054008] ... event mask: 000000070000000f [ 0.056196] rcu: Hierarchical SRCU implementation. [ 0.058679] smp: Bringing up secondary CPUs ... [ 0.059656] x86: Booting SMP configuration: [ 0.060030] .... node #0, CPUs: #1 #2 #3 [ 0.071192] smp: Brought up 1 node, 4 CPUs [ 0.073012] smpboot: Max logical packages: 1 [ 0.074017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.111022] node 0 deferred pages initialised in 35ms [ 0.114467] devtmpfs: initialized [ 0.115275] x86/mm: Memory block size: 128MB [ 0.117755] gcov: version magic: 0x41383552 [ 0.121418] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.124207] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128449] pinctrl core: initialized pinctrl subsystem [ 0.131456] [ 0.136081] ************************************************************* [ 0.139011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146013] ** ** [ 0.152016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158014] ** ** [ 0.162011] ** This means that this kernel is built to expose internal ** [ 0.164013] ** IOMMU data structures, which may compromise security on ** [ 0.167097] ** your system. ** [ 0.169016] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.175016] ** kernel, report this immediately to your vendor! ** [ 0.176010] ** ** [ 0.178013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180009] ************************************************************* [ 0.182921] NET: Registered protocol family 16 [ 0.185554] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196708] cpuidle: using governor menu [ 0.197635] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200594] PCI: Using configuration type 1 for base access [ 0.203121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214583] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218011] cryptd: max_cpu_qlen set to 1000 [ 0.221314] ACPI: Added _OSI(Module Device) [ 0.222000] ACPI: Added _OSI(Processor Device) [ 0.223017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231786] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238390] ACPI: Interpreter enabled [ 0.240038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241007] ACPI: Using IOAPIC for interrupt routing [ 0.242101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246667] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258390] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277381] acpiphp: Slot [2] registered [ 0.279148] acpiphp: Slot [5] registered [ 0.280131] acpiphp: Slot [6] registered [ 0.283046] acpiphp: Slot [7] registered [ 0.284179] acpiphp: Slot [8] registered [ 0.287260] acpiphp: Slot [9] registered [ 0.290160] acpiphp: Slot [10] registered [ 0.292124] acpiphp: Slot [3] registered [ 0.294087] acpiphp: Slot [4] registered [ 0.295203] acpiphp: Slot [11] registered [ 0.297097] acpiphp: Slot [12] registered [ 0.299085] acpiphp: Slot [13] registered [ 0.301172] acpiphp: Slot [14] registered [ 0.302175] acpiphp: Slot [15] registered [ 0.304086] acpiphp: Slot [16] registered [ 0.306077] acpiphp: Slot [17] registered [ 0.308178] acpiphp: Slot [18] registered [ 0.310102] acpiphp: Slot [19] registered [ 0.311156] acpiphp: Slot [20] registered [ 0.313065] acpiphp: Slot [21] registered [ 0.314086] acpiphp: Slot [22] registered [ 0.316070] acpiphp: Slot [23] registered [ 0.317094] acpiphp: Slot [24] registered [ 0.319129] acpiphp: Slot [25] registered [ 0.321070] acpiphp: Slot [26] registered [ 0.322138] acpiphp: Slot [27] registered [ 0.324083] acpiphp: Slot [28] registered [ 0.325087] acpiphp: Slot [29] registered [ 0.327098] acpiphp: Slot [30] registered [ 0.329118] acpiphp: Slot [31] registered [ 0.331076] PCI host bridge to bus 0000:00 [ 0.333020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351233] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.354185] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358432] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370985] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.381014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.384015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390622] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393764] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.398751] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.415015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.422016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.428981] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.436023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.445020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466967] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.490017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.521000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.527000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.536015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.542046] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.560037] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.583000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.591089] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.616034] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.639546] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.648018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.710082] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.720604] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.736022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.755019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.770000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.774934] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.779605] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.783440] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.786247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.791074] iommu: Default domain type: Passthrough [ 0.793564] SCSI subsystem initialized [ 0.795132] ACPI: bus type USB registered [ 0.797134] usbcore: registered new interface driver usbfs [ 0.799065] usbcore: registered new interface driver hub [ 0.801091] usbcore: registered new device driver usb [ 0.803160] pps_core: LinuxPPS API ver. 1 registered [ 0.804009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.808111] PTP clock support registered [ 0.810506] EDAC MC: Ver: 3.0.0 [ 0.813670] PCI: Using ACPI for IRQ routing [ 0.816034] NetLabel: Initializing [ 0.817010] NetLabel: domain hash size = 128 [ 0.820011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.822077] NetLabel: unlabeled traffic allowed by default [ 0.824914] vgaarb: loaded [ 0.827283] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.828009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.838193] clocksource: Switched to clocksource kvm-clock [ 0.974292] VFS: Disk quotas dquot_6.6.0 [ 0.979196] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.984199] *** VALIDATE ramfs *** [ 0.986040] *** VALIDATE hugetlbfs *** [ 0.988504] pnp: PnP ACPI init [ 0.991579] pnp: PnP ACPI: found 6 devices [ 1.015419] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.020387] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.023441] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.026470] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.029493] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.032174] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.035319] NET: Registered protocol family 2 [ 1.037933] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.044131] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.047776] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.053599] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.058403] TCP: Hash tables configured (established 65536 bind 65536) [ 1.061798] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.065390] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069328] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.073082] NET: Registered protocol family 1 [ 1.077354] RPC: Registered named UNIX socket transport module. [ 1.080380] RPC: Registered udp transport module. [ 1.082530] RPC: Registered tcp transport module. [ 1.084618] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.087319] NET: Registered protocol family 44 [ 1.089570] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.092740] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.095495] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.098480] PCI: CLS 0 bytes, default 64 [ 1.100598] Unpacking initramfs... [ 3.042469] debug: unmapping init [mem 0xffff99197cc54000-0xffff99197ffbffff] [ 3.046889] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.049566] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.053077] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.659465] Initialise system trusted keyrings [ 3.661078] Key type blacklist registered [ 3.663401] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.673107] zbud: loaded [ 3.676216] *** VALIDATE nfs *** [ 3.677414] *** VALIDATE nfs4 *** [ 3.679362] pstore: using deflate compression [ 3.683464] Platform Keyring initialized [ 3.808635] NET: Registered protocol family 38 [ 3.810981] Key type asymmetric registered [ 3.812909] Asymmetric key parser 'x509' registered [ 3.815419] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.819304] io scheduler mq-deadline registered [ 3.820915] io scheduler kyber registered [ 3.826286] io scheduler bfq registered [ 3.828472] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.832757] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.836279] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.839577] ACPI: Power Button [PWRF] [ 3.845678] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.854911] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.898957] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.917817] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.959775] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.991109] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.019328] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.026734] Non-volatile memory driver v1.3 [ 4.028512] Linux agpgart interface v0.103 [ 4.070534] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 4.075317] vda: detected capacity change from 0 to 74862592 [ 4.131821] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.134873] vdb: detected capacity change from 0 to 1073741824 [ 4.152207] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.155121] vdc: detected capacity change from 0 to 2621440000 [ 4.172521] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.175444] vdd: detected capacity change from 0 to 2621440000 [ 4.189923] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.193099] vde: detected capacity change from 0 to 4294967296 [ 4.212501] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.215994] vdf: detected capacity change from 0 to 4294967296 [ 4.224599] libphy: Fixed MDIO Bus: probed [ 4.231393] usbcore: registered new interface driver usbserial_generic [ 4.234078] usbserial: USB Serial support registered for generic [ 4.236839] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.241363] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.243760] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.246336] mousedev: PS/2 mouse device common for all mice [ 4.249976] rtc_cmos 00:05: RTC can wake from S4 [ 4.254784] rtc_cmos 00:05: registered as rtc0 [ 4.257103] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.261580] intel_pstate: CPU model not supported [ 4.265578] hid: raw HID events driver (C) Jiri Kosina [ 4.268400] usbcore: registered new interface driver usbhid [ 4.269894] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.272823] usbhid: USB HID core driver [ 4.273872] drop_monitor: Initializing network drop monitor service [ 4.288296] Initializing XFRM netlink socket [ 4.291767] NET: Registered protocol family 10 [ 4.296819] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.300495] Segment Routing with IPv6 [ 4.304224] NET: Registered protocol family 17 [ 4.307087] mpls_gso: MPLS GSO support [ 4.312608] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.321343] RAS: Correctable Errors collector initialized. [ 4.324192] AVX version of gcm_enc/dec engaged. [ 4.326796] AES CTR mode by8 optimization enabled [ 4.502277] sched_clock: Marking stable (4502241933, 0)->(5531270549, -1029028616) [ 4.508541] registered taskstats version 1 [ 4.510386] Loading compiled-in X.509 certificates [ 4.511934] zswap: loaded using pool lzo/zbud [ 4.539717] Key type big_key registered [ 4.553463] Key type encrypted registered [ 4.555233] ima: No TPM chip found, activating TPM-bypass! [ 4.557758] ima: Allocated hash algorithm: sha1 [ 4.559354] ima: No architecture policies found [ 4.563810] evm: Initialising EVM extended attributes: [ 4.567863] evm: security.selinux [ 4.571381] evm: security.ima [ 4.573254] evm: security.capability [ 4.574635] evm: HMAC attrs: 0x1 [ 4.577450] rtc_cmos 00:05: setting system clock to 2026-08-24 05:09:10 UTC (1787548150) [ 4.585170] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 4.587848] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 4.599589] Write protecting the kernel read-only data: 28672k [ 4.602830] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 4.605216] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 4.646530] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.658689] systemd[1]: Detected virtualization kvm. [ 4.664661] systemd[1]: Detected architecture x86-64. [ 4.666722] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.694594] systemd[1]: No hostname configured. [ 4.698789] systemd[1]: Set hostname to . [ 4.702840] random: systemd: uninitialized urandom read (16 bytes read) [ 4.705521] systemd[1]: Initializing machine ID from random generator. [ 4.796903] random: ln: uninitialized urandom read (6 bytes read) [ 4.932813] random: systemd: uninitialized urandom read (16 bytes read) [ 4.986381] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.993904] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.004924] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.706439] device-mapper: uevent: version 1.0.3 [ 6.712976] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.223654] random: fast init done [ 9.409950] virtio_net virtio0 ens2: renamed from eth0 [ 11.223295] scsi host0: ata_piix [ 11.388324] scsi host1: ata_piix [ 11.390130] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.472949] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.500153] random: crng init done [ 16.502414] random: 7 urandom warning(s) missed due to ratelimiting [ 20.153562] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.313627] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.119186] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.644308] SELinux: Disabled at runtime. [ 25.844939] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.856266] systemd[1]: Detected virtualization kvm. [ 25.858079] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.323933] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.329535] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.377424] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.381817] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.385588] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.394346] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.470415] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 29.364642] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.900397] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.595413] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.233193] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.680299] hrtimer: interrupt took 5694827 ns [ 35.237446] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 37.157994] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 42.756163] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (15s / no limit)[ 43.663864] NFS: Registering the id_resolver key type [ 43.672524] Key type id_resolver registered [ 43.674877] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg301-server login: [ 99.213349] libcfs: loading out-of-tree module taints kernel. [ 99.243687] Key type ._llcrypt registered [ 99.245728] Key type .llcrypt registered [ 99.319347] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 115.735774] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 118.389804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.412786] alg: No test for adler32 (adler32-zlib) [ 120.216970] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 121.754796] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 123.993354] Key type lgssc registered [ 125.616536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.170259] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 195.180961] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 208.492785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.528359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.792477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.836838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.925550] Lustre: lustre-MDT0000: new disk, initializing [ 210.034400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.077375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.699478] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.012024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.129331] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 229.171859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 229.177384] Lustre: Skipped 1 previous similar message [ 229.333178] Lustre: lustre-MDT0001: new disk, initializing [ 229.431296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.462368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 229.475550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.090184] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.707418] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.447765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.889994] Lustre: lustre-OST0000: new disk, initializing [ 248.912611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.922968] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.983565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.890142] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.208912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.229184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.356678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 274.922278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 275.022784] Lustre: lustre-OST0001: new disk, initializing [ 275.032957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 275.043847] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 275.095996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.442796] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.219146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 285.228799] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 285.380392] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.650965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.604374] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 311.639712] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing check_logdir /tmp/testlogs/ [ 316.635686] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing yml_node [ 320.618258] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 323.042382] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 325.663135] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 327.381880] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 01:14:31 EDT 2026 [ 342.625070] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 343.915263] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.952380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.953912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.964016] Lustre: Skipped 2 previous similar messages [ 362.011499] Lustre: Skipped 3 previous similar messages [ 366.063579] Lustre: server umount lustre-MDT0000 complete [ 369.576891] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548515 with bad export cookie 10818591310800434392 [ 369.579470] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.583710] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 372.196061] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.197403] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.210309] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 372.223489] Lustre: Skipped 1 previous similar message [ 372.230533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.233699] Lustre: Skipped 1 previous similar message [ 374.240515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.240547] LustreError: 8672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.257939] LustreError: 8672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 376.278479] Lustre: server umount lustre-MDT0001 complete [ 379.668161] Lustre: server umount lustre-OST0000 complete [ 383.776754] Lustre: server umount lustre-OST0001 complete [ 390.758722] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 400.116752] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 435.247331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.512430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 435.546532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 435.600623] Lustre: lustre-MDT0000: new disk, initializing [ 435.686092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.709698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 440.125752] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 449.558487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.691442] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.781057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.787181] Lustre: Skipped 1 previous similar message [ 449.875474] Lustre: lustre-MDT0001: new disk, initializing [ 449.983050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.029518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.039594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.787694] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 462.504369] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 469.042563] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 475.916429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.260990] Lustre: lustre-OST0000: new disk, initializing [ 476.265592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 476.285276] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 476.407120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.058068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 478.073630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 478.230417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.690560] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 493.200693] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 497.801387] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.040343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.286223] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 502.498291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 508.896632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.916284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.931249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.943413] Lustre: Skipped 2 previous similar messages [ 508.958658] Lustre: Skipped 2 previous similar messages [ 510.985840] Lustre: server umount lustre-OST0000 complete [ 516.578610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 516.581973] LustreError: Skipped 1 previous similar message [ 516.588147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.596889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.361472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.376969] Lustre: Skipped 5 previous similar messages [ 529.375191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 529.604660] Lustre: server umount lustre-MDT0000 complete [ 533.171598] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548679 with bad export cookie 10818591310800435918 [ 533.186627] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.187464] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 533.555883] Lustre: server umount lustre-MDT0001 complete [ 537.187476] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:18:00 (1787548680) [ 543.739706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.115928] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.191376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.819554] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 549.345250] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.470712] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.067037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.666590] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 572.223440] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 578.334343] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.566948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.502497] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 600.497165] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.823156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 611.838073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.845049] Lustre: Skipped 1 previous similar message [ 611.851789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 613.876455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 617.077804] Lustre: server umount lustre-OST0000 complete [ 624.096163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.107918] Lustre: Skipped 2 previous similar messages [ 635.359954] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 635.537464] Lustre: server umount lustre-MDT0000 complete [ 638.433491] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.456587] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 639.608660] LustreError: 19304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548785 with bad export cookie 10818591310800436807 [ 639.613183] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.623223] LustreError: 19304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 640.085532] Lustre: server umount lustre-MDT0001 complete [ 652.948932] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 656.125726] Key type lgssc unregistered [ 656.463250] LNet: 22862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.474835] LNetError: 22862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.486870] LNet: Removed LNI 192.168.203.101@tcp [ 657.475168] Key type .llcrypt unregistered [ 657.484318] Key type ._llcrypt unregistered [ 683.303734] Key type ._llcrypt registered [ 683.307338] Key type .llcrypt registered [ 683.495776] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 684.847825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 684.921266] alg: No test for adler32 (adler32-zlib) [ 686.178554] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 686.609571] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 688.383203] Key type lgssc registered [ 690.100581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 697.509641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.553443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.918766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.188993] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 709.092516] LustreError: 24278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.113881] LustreError: 24278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 710.055278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.656187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.357355] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 723.141419] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 728.513725] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 735.775507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.193258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.386938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 745.173048] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 755.088526] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 775.141039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 775.159954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.179957] LustreError: 26101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 775.422781] Lustre: server umount lustre-OST0000 complete [ 781.285745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.289536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.311596] Lustre: Skipped 1 previous similar message [ 781.329813] Lustre: Skipped 1 previous similar message [ 786.400938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.429611] Lustre: Skipped 1 previous similar message [ 786.665896] Lustre: server umount lustre-MDT0000 complete [ 791.520872] LustreError: 25623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.549096] LustreError: 25623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 792.543520] Lustre: 23592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787548922/real 1787548922] req@00000000394f9a58 x1874380804096000/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787548938 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 792.547319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.596827] Lustre: 23592:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 793.174759] LustreError: 24261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548939 with bad export cookie 2476413572288435012 [ 793.198724] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.208248] LustreError: 24261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 793.758475] Lustre: server umount lustre-MDT0001 complete [ 812.957789] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 816.934941] Key type lgssc unregistered [ 817.167770] LNet: 28280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.177248] LNetError: 28280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.203302] LNet: Removed LNI 192.168.203.101@tcp [ 818.182334] Key type .llcrypt unregistered [ 818.188715] Key type ._llcrypt unregistered [ 838.450823] Key type ._llcrypt registered [ 838.454032] Key type .llcrypt registered [ 838.594894] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 839.480785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.499357] alg: No test for adler32 (adler32-zlib) [ 840.435765] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 840.664954] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 842.314092] Key type lgssc registered [ 843.245514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.667204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 851.690092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.358255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.521198] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 863.713102] LustreError: 29673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.751213] LustreError: 29673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 867.804285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.378417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.712757] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 886.315157] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 892.318826] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 905.903353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.400444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.888749] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 923.978307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 925.496473] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 934.373378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.378293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.378378] LustreError: Skipped 1 previous similar message [ 934.385505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.390908] Lustre: Skipped 1 previous similar message [ 937.305520] Lustre: server umount lustre-OST0000 complete [ 944.609605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.614309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.632358] Lustre: Skipped 2 previous similar messages [ 944.637792] Lustre: Skipped 1 previous similar message [ 949.729204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.736781] Lustre: Skipped 1 previous similar message [ 954.854652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.859492] Lustre: Skipped 1 previous similar message [ 955.359153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 955.558782] Lustre: server umount lustre-MDT0000 complete [ 958.908160] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549104 with bad export cookie 9887487434204908532 [ 958.911184] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.922055] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 959.240311] Lustre: server umount lustre-MDT0001 complete [ 968.894045] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 971.304600] Key type lgssc unregistered [ 971.600260] LNet: 33557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.617992] LNetError: 33557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.638849] LNet: Removed LNI 192.168.203.101@tcp [ 972.528863] Key type .llcrypt unregistered [ 972.532072] Key type ._llcrypt unregistered [ 994.961903] Key type ._llcrypt registered [ 994.965404] Key type .llcrypt registered [ 995.113041] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 996.145233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.341708] alg: No test for adler32 (adler32-zlib) [ 997.461127] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 997.801294] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 999.488170] Key type lgssc registered [ 1000.679667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.273223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.285202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.752242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.472627] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1019.876415] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.904764] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1022.599361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.914944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.144571] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1033.668188] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1038.366918] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1044.325794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.639655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.292956] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1054.936576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1058.818761] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1068.513420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1068.520603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.546091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.038979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1069.046944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.058348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.567683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.573680] Lustre: Skipped 1 previous similar message [ 1074.292817] Lustre: server umount lustre-OST0000 complete [ 1079.264458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1079.276493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.292961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.920174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.928303] Lustre: Skipped 3 previous similar messages [ 1092.075088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1092.223791] Lustre: server umount lustre-MDT0000 complete [ 1095.137558] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.170516] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1095.639024] LustreError: 34951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549241 with bad export cookie 3024612150173402206 [ 1095.653049] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.658977] LustreError: 34951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.912510] Lustre: server umount lustre-MDT0001 complete [ 1105.658581] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1108.289456] Key type lgssc unregistered [ 1108.646305] LNet: 38508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.656942] LNetError: 38508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.674672] LNet: Removed LNI 192.168.203.101@tcp [ 1109.547457] Key type .llcrypt unregistered [ 1109.550266] Key type ._llcrypt unregistered [ 1129.559876] Key type ._llcrypt registered [ 1129.564388] Key type .llcrypt registered [ 1129.680596] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1130.975847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1131.083566] alg: No test for adler32 (adler32-zlib) [ 1132.295888] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1132.635629] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1134.359548] Key type lgssc registered [ 1135.395691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.240413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.274901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.809758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.245707] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1156.069164] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.091270] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1158.634955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.987149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.616212] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1168.496530] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1173.909030] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1179.543046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.844386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.101931] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1189.913258] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1190.133643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1197.734459] Lustre: lustre-OST0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1199.585871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1199.593626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.602977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.610251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.610823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1200.616330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.621535] Lustre: Skipped 1 previous similar message [ 1202.836108] Lustre: lustre-OST0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1207.969380] Lustre: lustre-OST0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1207.981114] Lustre: Skipped 2 previous similar messages [ 1210.335163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1210.459047] Lustre: server umount lustre-OST0000 complete [ 1230.818766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.825810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.842347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.427741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.022150] Lustre: server umount lustre-MDT0000 complete [ 1241.156813] LustreError: 39880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549387 with bad export cookie 6633893747381982297 [ 1241.167391] LustreError: 39880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1241.167438] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.393831] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1255.816294] Lustre: server umount lustre-MDT0001 complete [ 1266.692314] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1270.583373] Key type lgssc unregistered [ 1270.888582] LNet: 43615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.901034] LNetError: 43615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.926314] LNet: Removed LNI 192.168.203.101@tcp [ 1272.105337] Key type .llcrypt unregistered [ 1272.110475] Key type ._llcrypt unregistered [ 1305.971089] Key type ._llcrypt registered [ 1305.974725] Key type .llcrypt registered [ 1306.057076] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1307.069510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.164132] alg: No test for adler32 (adler32-zlib) [ 1308.302852] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1308.630472] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1310.383199] Key type lgssc registered [ 1311.548354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1319.255815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1319.272847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.760803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.339290] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1330.166203] LustreError: 45027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.198908] LustreError: 45027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1335.272396] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.551394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.511616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.756701] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1352.478187] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.167775] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.434390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.807218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.833748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1370.606365] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1377.432087] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1387.492863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.496174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1387.505373] Lustre: Skipped 1 previous similar message [ 1389.247803] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 1389.252109] Lustre: Skipped 2 previous similar messages [ 1392.614911] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.636233] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1392.685167] Lustre: server umount lustre-MDT0000 complete [ 1396.173401] LustreError: 45009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549542 with bad export cookie 11372445413915812985 [ 1396.175695] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.194406] LustreError: 45009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1397.728918] LustreError: 46136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.730610] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.762491] LustreError: 46136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1397.792084] Lustre: Skipped 1 previous similar message [ 1397.798753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1401.760102] LustreError: 46136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.761940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1401.772408] LustreError: 46136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1401.784169] Lustre: Skipped 1 previous similar message [ 1402.800995] Lustre: server umount lustre-MDT0001 complete [ 1417.717764] Lustre: server umount lustre-OST0000 complete [ 1433.576304] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1435.856617] Key type lgssc unregistered [ 1436.110839] LNet: 48890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.115225] LNetError: 48890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.139846] LNet: Removed LNI 192.168.203.101@tcp [ 1436.862134] Key type .llcrypt unregistered [ 1436.863883] Key type ._llcrypt unregistered [ 1462.919211] Key type ._llcrypt registered [ 1462.921655] Key type .llcrypt registered [ 1463.016154] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1464.280375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.307728] alg: No test for adler32 (adler32-zlib) [ 1465.337617] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1465.525861] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1467.175175] Key type lgssc registered [ 1468.087767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1475.508382] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1475.548661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.167427] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1496.287438] LustreError: 50281:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 1522.912483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.023207] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1534.257758] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1700.569144] Lustre: server umount lustre-OST0000 complete [ 1715.149635] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1717.519420] Key type lgssc unregistered [ 1717.805812] LNet: 51958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.816154] LNetError: 51958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.865512] LNet: Removed LNI 192.168.203.101@tcp [ 1718.862305] Key type .llcrypt unregistered [ 1718.865259] Key type ._llcrypt unregistered [ 1742.295204] Key type ._llcrypt registered [ 1742.298649] Key type .llcrypt registered [ 1742.415674] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1743.263783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.282899] alg: No test for adler32 (adler32-zlib) [ 1744.360435] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1744.641326] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1746.415211] Key type lgssc registered [ 1747.656548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.699344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.723218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.514841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.359157] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1767.905356] LustreError: 53372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1767.919154] LustreError: 53372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1772.468895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.747287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1778.351634] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1785.617428] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1792.009217] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1798.657669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1799.226957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.303050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1806.857667] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1816.301850] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1822.703082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1822.708089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.723495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.789695] Lustre: Skipped 1 previous similar message [ 1826.437897] Lustre: server umount lustre-OST0000 complete [ 1832.930771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.957369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1832.961359] Lustre: Skipped 1 previous similar message [ 1832.983411] Lustre: Skipped 2 previous similar messages [ 1837.680200] Lustre: server umount lustre-MDT0000 complete [ 1838.052633] LustreError: 53371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.086276] LustreError: 53371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1841.696539] LustreError: 53353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549987 with bad export cookie 7334979580890982255 [ 1841.700787] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1841.732112] LustreError: 53353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1842.309806] Lustre: server umount lustre-MDT0001 complete [ 1852.547737] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1855.129233] Key type lgssc unregistered [ 1855.369675] LNet: 56909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.386320] LNetError: 56909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.407734] LNet: Removed LNI 192.168.203.101@tcp [ 1856.007718] Key type .llcrypt unregistered [ 1856.010482] Key type ._llcrypt unregistered [ 1886.523723] Key type ._llcrypt registered [ 1886.529646] Key type .llcrypt registered [ 1886.645653] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1887.899927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1887.952648] alg: No test for adler32 (adler32-zlib) [ 1888.919538] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1889.177204] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1890.839170] Key type lgssc registered [ 1892.126178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.084878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.098756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1917.728526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1922.850171] LustreError: 58298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 1949.471455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.583352] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1962.533906] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.059546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.183523] LustreError: 57633:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000644ef482 x1874382065502464/t0(0) o250->MGC192.168.203.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.749355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.844276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1974.415184] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1974.761460] LustreError: 58323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.791944] LustreError: 58323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1979.892948] LustreError: 58324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.789252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.101877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.626799] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1994.642915] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.087480] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2003.424394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.428285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.465489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.968644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.974305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.991135] Lustre: Skipped 1 previous similar message [ 2008.436876] Lustre: server umount lustre-OST0000 complete [ 2034.144712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2034.172275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.206576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.085793] Lustre: server umount lustre-MDT0000 complete [ 2040.553910] LustreError: 58305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550186 with bad export cookie 14137938504006331780 [ 2040.578123] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.916490] Lustre: server umount lustre-MDT0001 complete [ 2056.525304] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2059.235036] Key type lgssc unregistered [ 2059.507632] LNet: 62047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.515040] LNetError: 62047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.534463] LNet: Removed LNI 192.168.203.101@tcp [ 2060.446230] Key type .llcrypt unregistered [ 2060.450297] Key type ._llcrypt unregistered [ 2082.316709] Key type ._llcrypt registered [ 2082.318184] Key type .llcrypt registered [ 2082.381164] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2083.723185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.857613] alg: No test for adler32 (adler32-zlib) [ 2084.980539] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2085.178757] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2086.911218] Key type lgssc registered [ 2088.286121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2097.008896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2097.023391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.454863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.628442] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2108.900590] LustreError: 63457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.912640] LustreError: 63457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2111.202140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.840895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.528275] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2128.573866] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2136.366601] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2145.207291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.825420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.624498] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2159.940562] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2168.302094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2177.505792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2177.511489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.522087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.558167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.562276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.581667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.647104] Lustre: server umount lustre-OST0000 complete [ 2188.771157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.773498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.784554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.802027] Lustre: Skipped 1 previous similar message [ 2188.805795] Lustre: Skipped 1 previous similar message [ 2193.268799] Lustre: server umount lustre-MDT0000 complete [ 2193.889932] LustreError: 64187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.909787] LustreError: 64187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2196.638074] LustreError: 63440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550342 with bad export cookie 3385992939876994410 [ 2196.640501] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.649288] LustreError: 63440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2196.943239] Lustre: server umount lustre-MDT0001 complete [ 2205.599071] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2207.955333] Key type lgssc unregistered [ 2208.267668] LNet: 67021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.272792] LNetError: 67021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.288812] LNet: Removed LNI 192.168.203.101@tcp [ 2209.082727] Key type .llcrypt unregistered [ 2209.086158] Key type ._llcrypt unregistered [ 2232.252761] Key type ._llcrypt registered [ 2232.255913] Key type .llcrypt registered [ 2232.412535] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2234.407685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.476664] alg: No test for adler32 (adler32-zlib) [ 2235.632844] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2235.990668] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2237.807176] Key type lgssc registered [ 2239.289583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.420128] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:46:35 (1787550395) [ 2261.492868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2261.513501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.093683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.825737] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2272.232504] LustreError: 68725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.305706] LustreError: 68725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2277.350706] LustreError: 68726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.503376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.745519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.780325] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2292.213050] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2300.020954] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2307.316045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2307.816045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.863189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2315.254576] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2323.585281] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2331.690912] Lustre: Failing over lustre-MDT0000 [ 2331.869851] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 2332.166706] Lustre: server umount lustre-MDT0000 complete [ 2333.665892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.669719] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.678516] Lustre: Skipped 2 previous similar messages [ 2333.709037] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2336.920542] LustreError: 68722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.044654] LustreError: 68722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.080510] LustreError: 68722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2342.946615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.082805] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.261853] Lustre: *** cfs_fail_loc=135, val=0*** [ 2343.429307] LustreError: 71814:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2343.446928] LustreError: 71814:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -2 [ 2343.455269] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2343.455269] [ 2343.463776] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2343.487262] LustreError: 71802:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2343.492720] LustreError: 71802:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2343.503482] LustreError: 71802:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2343.518295] LustreError: 71802:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2343.551530] Lustre: server umount lustre-MDT0000 complete [ 2343.554638] LustreError: 71802:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2350.390464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.492770] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.512327] LustreError: 72160:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.101@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000767e613e) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2350.528100] LustreError: 68717:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2350.811892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.865968] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2352.274602] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2355.181469] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2356.204401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2356.254701] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2356.328929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2364.388653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2364.398150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.415574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.439830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.443586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.459233] Lustre: Skipped 1 previous similar message [ 2369.765767] Lustre: server umount lustre-OST0000 complete [ 2376.674476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.687348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2376.690674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.696503] Lustre: Skipped 1 previous similar message [ 2380.491902] Lustre: server umount lustre-MDT0000 complete [ 2381.795108] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.817303] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2384.228869] LustreError: 68705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550530 with bad export cookie 15972888969287256631 [ 2384.230617] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.237307] LustreError: 68705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2384.702421] Lustre: server umount lustre-MDT0001 complete [ 2396.140213] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2399.151896] Key type lgssc unregistered [ 2399.415504] LNet: 73596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.421842] LNetError: 73596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.447071] LNet: Removed LNI 192.168.203.101@tcp [ 2400.218649] Key type .llcrypt unregistered [ 2400.223451] Key type ._llcrypt unregistered [ 2428.250691] Key type ._llcrypt registered [ 2428.254027] Key type .llcrypt registered [ 2428.318521] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2429.429752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2429.454873] alg: No test for adler32 (adler32-zlib) [ 2430.748115] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2431.006204] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2432.711114] Key type lgssc registered [ 2434.369557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2441.395936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2441.435244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.838390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.540136] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2452.545360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.078941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.358978] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2464.727524] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.929787] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2477.685887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.004213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.613541] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2487.238582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2492.296850] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2502.368398] Lustre: Failing over lustre-MDT0000 [ 2502.629369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.641443] Lustre: Skipped 1 previous similar message [ 2502.641760] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.661862] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2502.690832] Lustre: server umount lustre-MDT0000 complete [ 2504.392762] LustreError: 75006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.750511] LustreError: 77308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.758807] LustreError: 77308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2512.875179] LustreError: 75008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.906844] LustreError: 75008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2515.397738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.853228] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.223340] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2516.479709] LustreError: 78100:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2516.496550] LustreError: 78100:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -9 [ 2516.517661] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2516.517661] [ 2516.530548] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2516.553947] LustreError: 78089:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2516.582325] LustreError: 78089:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2516.593469] LustreError: 78089:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2516.597089] LustreError: 78089:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2516.610041] Lustre: server umount lustre-MDT0000 complete [ 2516.613113] LustreError: 78089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2517.990472] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.022582] LustreError: 75012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2523.608869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.763924] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.778797] LustreError: 75004:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2524.061843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.135899] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2524.812635] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2528.734099] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2529.266355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2529.316226] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2529.375199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2539.495859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.502041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2539.515933] Lustre: Skipped 2 previous similar messages [ 2539.534886] Lustre: Skipped 1 previous similar message [ 2543.615083] Lustre: server umount lustre-OST0000 complete [ 2549.731465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.734068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.750861] Lustre: Skipped 1 previous similar message [ 2549.778591] Lustre: Skipped 1 previous similar message [ 2553.813643] Lustre: server umount lustre-MDT0000 complete [ 2554.848055] LustreError: 75011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.872843] LustreError: 75011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2557.848411] LustreError: 74991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550703 with bad export cookie 14618060645241543034 [ 2557.853881] LustreError: 74991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2557.857120] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.366669] Lustre: server umount lustre-MDT0001 complete [ 2569.362974] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2572.210651] Key type lgssc unregistered [ 2572.519584] LNet: 79885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2572.528841] LNetError: 79885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.541838] LNet: Removed LNI 192.168.203.101@tcp [ 2574.596426] Key type .llcrypt unregistered [ 2574.599974] Key type ._llcrypt unregistered [ 2587.900917] Key type ._llcrypt registered [ 2587.902622] Key type .llcrypt registered [ 2588.027583] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2588.321311] Key type .llcrypt unregistered [ 2588.323953] Key type ._llcrypt unregistered [ 2609.117931] Key type ._llcrypt registered [ 2609.121062] Key type .llcrypt registered [ 2609.226377] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2610.059314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.116759] alg: No test for adler32 (adler32-zlib) [ 2611.205623] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2611.465699] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2613.135203] Key type lgssc registered [ 2614.105953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.362626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.383130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.768344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.062840] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2631.139321] LustreError: 81793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.158532] LustreError: 81793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2633.086363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.564196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2639.134612] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2646.452518] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.955724] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2663.436307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.170729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.232435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2675.280493] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2687.515488] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2697.191405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2697.197620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.217866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.234857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.236426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.243106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.257980] Lustre: Skipped 1 previous similar message [ 2702.440367] Lustre: server umount lustre-OST0000 complete [ 2709.483303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.536725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.554658] Lustre: Skipped 1 previous similar message [ 2713.082917] Lustre: server umount lustre-MDT0000 complete [ 2714.596595] LustreError: 82521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.632922] LustreError: 82521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2717.984622] LustreError: 81774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550863 with bad export cookie 2227816083213780606 [ 2718.004770] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.586038] Lustre: server umount lustre-MDT0001 complete [ 2730.834305] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2734.470296] Key type lgssc unregistered [ 2734.767171] LNet: 85331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.773988] LNetError: 85331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.789551] LNet: Removed LNI 192.168.203.101@tcp [ 2735.520593] Key type .llcrypt unregistered [ 2735.521990] Key type ._llcrypt unregistered [ 2757.682111] Key type ._llcrypt registered [ 2757.686162] Key type .llcrypt registered [ 2757.789428] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2758.782613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.813839] alg: No test for adler32 (adler32-zlib) [ 2759.797335] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2760.065329] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2761.736828] Key type lgssc registered [ 2762.934263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2770.570615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2770.612667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.109118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.777073] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2781.667583] LustreError: 86752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.696632] LustreError: 86752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2785.738886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.276605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.198367] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2798.791205] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2804.685948] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2811.552941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.822724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.873375] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2816.939952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2822.720453] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2831.370852] Lustre: server umount lustre-OST0000 complete [ 2837.474585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.479331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2837.482219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2837.500817] Lustre: Skipped 1 previous similar message [ 2837.536437] Lustre: Skipped 1 previous similar message [ 2842.598761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.603231] Lustre: Skipped 1 previous similar message [ 2845.721398] Lustre: server umount lustre-MDT0000 complete [ 2847.723176] LustreError: 86747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.752237] LustreError: 86747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2848.738693] Lustre: 86064:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550978/real 1787550978] req@000000000989ed24 x1874382977786496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787550994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2848.793205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.828566] LustreError: 88594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.837063] LustreError: 88594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2849.492372] LustreError: 86735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550995 with bad export cookie 13052473484499576890 [ 2849.502746] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.504856] LustreError: 86735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2849.926986] Lustre: server umount lustre-MDT0001 complete [ 2860.056986] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2862.417458] Key type lgssc unregistered [ 2862.726815] LNet: 90288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.743959] LNetError: 90288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.781485] LNet: Removed LNI 192.168.203.101@tcp [ 2863.656418] Key type .llcrypt unregistered [ 2863.660349] Key type ._llcrypt unregistered [ 2888.356747] Key type ._llcrypt registered [ 2888.366136] Key type .llcrypt registered [ 2888.494949] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2889.751649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2889.884975] alg: No test for adler32 (adler32-zlib) [ 2890.998747] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2891.244703] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2892.967229] Key type lgssc registered [ 2894.216448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2904.028543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2904.068389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.617098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.194678] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2915.820281] LustreError: 91702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.851771] LustreError: 91702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2919.980317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.384777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2923.878439] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2931.327795] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2940.886372] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2947.465317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.765426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.840048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2953.202264] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2961.239170] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2981.861740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2981.891091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.916625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2981.929396] Lustre: Skipped 1 previous similar message [ 2983.488465] Lustre: server umount lustre-OST0000 complete [ 2991.080306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.111315] Lustre: Skipped 1 previous similar message [ 2991.120871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.139957] Lustre: Skipped 1 previous similar message [ 2996.880347] Lustre: server umount lustre-MDT0000 complete [ 2997.222254] LustreError: 91703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.242970] LustreError: 91703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3000.866984] LustreError: 91684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551146 with bad export cookie 3655195573062970738 [ 3000.872688] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.874748] LustreError: 91684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3001.106077] Lustre: server umount lustre-MDT0001 complete [ 3012.022828] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3016.014937] Key type lgssc unregistered [ 3016.398730] LNet: 95262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.407092] LNetError: 95262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3017.456275] LNet: Removed LNI 192.168.203.101@tcp [ 3018.821298] Key type .llcrypt unregistered [ 3018.824781] Key type ._llcrypt unregistered [ 3041.709764] Key type ._llcrypt registered [ 3041.711576] Key type .llcrypt registered [ 3041.811499] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3043.318805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3043.481896] alg: No test for adler32 (adler32-zlib) [ 3044.604632] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 3044.964132] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3046.625160] Key type lgssc registered [ 3047.919773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3057.659936] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3057.681351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3073.254492] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3078.367296] LustreError: 96629:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 3105.055733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.884552] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3120.847031] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3126.197630] Lustre: server umount lustre-OST0000 complete [ 3137.066901] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:01:20 (1787551280) [ 3146.731349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.736799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.866780] LustreError: 98284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.892209] LustreError: 98284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3154.138158] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3157.985908] LustreError: 98283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.453900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.086416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3167.306516] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3173.730868] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3179.217413] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3186.346324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3186.750595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.881560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3193.559534] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3201.305391] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3213.292893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.299897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3213.302704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3213.308316] Lustre: Skipped 1 previous similar message [ 3213.353585] LustreError: Skipped 1 previous similar message [ 3218.380507] Lustre: server umount lustre-OST0000 complete [ 3223.522902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.530108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.537531] Lustre: Skipped 1 previous similar message [ 3223.549856] Lustre: Skipped 2 previous similar messages [ 3228.642560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.653046] Lustre: Skipped 1 previous similar message [ 3233.770049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.781875] Lustre: Skipped 1 previous similar message [ 3236.319534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3236.907810] Lustre: server umount lustre-MDT0000 complete [ 3238.881865] LustreError: 98284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.906559] LustreError: 98284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3240.966343] LustreError: 98264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551386 with bad export cookie 9505144175130146173 [ 3240.999779] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.405726] Lustre: server umount lustre-MDT0001 complete [ 3251.604698] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3254.103822] Key type lgssc unregistered [ 3254.388437] LNet: 101983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.400137] LNetError: 101983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.429527] LNet: Removed LNI 192.168.203.101@tcp [ 3255.320418] Key type .llcrypt unregistered [ 3255.322725] Key type ._llcrypt unregistered [ 3297.767361] Key type ._llcrypt registered [ 3297.771201] Key type .llcrypt registered [ 3297.829115] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3298.532082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3298.544207] alg: No test for adler32 (adler32-zlib) [ 3299.741168] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 3299.976166] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3301.631185] Key type lgssc registered [ 3302.697328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3310.373316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3310.389969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.772464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.079053] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3321.315589] LustreError: 104169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.327849] LustreError: 104169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3324.020940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.489760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3329.149443] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3337.178991] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3342.984577] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3349.564427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.112119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.235902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3356.339914] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3365.984767] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3375.594094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.610950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.636573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3376.636182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3376.637950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.659574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3380.541709] Lustre: server umount lustre-OST0000 complete [ 3385.824411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3385.851528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.872238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.879548] Lustre: Skipped 1 previous similar message [ 3391.972876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.975844] Lustre: Skipped 3 previous similar messages [ 3397.095311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.135186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3399.277287] Lustre: server umount lustre-MDT0000 complete [ 3401.191232] LustreError: 104897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.245831] LustreError: 104897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3405.349866] LustreError: 104151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551551 with bad export cookie 1038152591388852703 [ 3405.359495] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.361638] LustreError: 104151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3405.806222] Lustre: server umount lustre-MDT0001 complete [ 3421.471613] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3425.533887] Key type lgssc unregistered [ 3425.944507] LNet: 107708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3425.976578] LNetError: 107708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3425.995957] LNet: Removed LNI 192.168.203.101@tcp [ 3427.199284] Key type .llcrypt unregistered [ 3427.201901] Key type ._llcrypt unregistered [ 3452.139330] Key type ._llcrypt registered [ 3452.142239] Key type .llcrypt registered [ 3452.241293] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3453.766986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3453.901498] alg: No test for adler32 (adler32-zlib) [ 3455.140691] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 3455.634372] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3457.327811] Key type lgssc registered [ 3458.662311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3466.776947] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3466.785979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.271377] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3487.391519] LustreError: 108950:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 3514.079254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.020695] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3532.842060] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3540.948983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.240786] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3541.262520] LustreError: 109975:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3541.269322] LustreError: 109975:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3541.277718] LustreError: 109975:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3541.285914] LustreError: 108958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551687 with bad export cookie 14888971864336640287 [ 3541.298577] LustreError: 108958:0:(ldlm_lock.c:2769:ldlm_lock_dump_handle()) ### ### ns: MGS lock: 0000000012c3af2b/0xcea0410684922949 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x40000000000000 nid: 0@lo remote: 0xcea0410684922942 expref: 2 pid: 109981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3541.325306] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3541.425500] Lustre: server umount lustre-MDT0000 complete [ 3541.432761] LustreError: 109975:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3549.987895] Lustre: server umount lustre-OST0000 complete [ 3559.046370] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 3569.934305] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3606.069637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.315561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3606.347297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3606.472314] Lustre: lustre-MDT0000: new disk, initializing [ 3606.541359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.555447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3611.246397] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3623.267938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.564558] Lustre: 112770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.654343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3623.659957] Lustre: Skipped 1 previous similar message [ 3623.760682] Lustre: lustre-MDT0001: new disk, initializing [ 3623.939334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3624.006867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3624.054454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3629.337858] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3639.308360] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3645.597400] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3653.250791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.570450] Lustre: lustre-OST0000: new disk, initializing [ 3653.579537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3653.596219] Lustre: 114736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3653.708847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3660.060084] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3662.970057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3662.985354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3663.170775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3670.276561] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3675.521186] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3675.701658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3680.472486] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3680.687694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3683.810146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.811411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.829470] LustreError: Skipped 1 previous similar message [ 3683.833664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.863724] Lustre: Skipped 1 previous similar message [ 3688.867519] Lustre: server umount lustre-OST0000 complete [ 3693.222487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.223588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3693.234430] Lustre: Skipped 1 previous similar message [ 3693.240736] Lustre: Skipped 1 previous similar message [ 3698.660931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.663883] Lustre: Skipped 2 previous similar messages [ 3699.703694] Lustre: server umount lustre-MDT0000 complete [ 3703.368056] LustreError: 112762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551849 with bad export cookie 14888971864336640385 [ 3703.368650] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.376751] LustreError: 112762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3703.586967] Lustre: server umount lustre-MDT0001 complete [ 3713.003657] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:10:56 (1787551856) [ 3718.455147] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3737.500320] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 3747.380071] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3780.598081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.801839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3780.839208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3780.937337] Lustre: lustre-MDT0000: new disk, initializing [ 3781.035164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.056228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3785.700288] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3796.682793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.800585] Lustre: 119639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3796.875986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3796.886953] Lustre: Skipped 1 previous similar message [ 3796.983899] Lustre: lustre-MDT0001: new disk, initializing [ 3797.153358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3797.203832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3797.212726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3802.306801] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3812.123240] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3817.934877] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3824.969573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.167939] Lustre: lustre-OST0000: new disk, initializing [ 3825.170769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3825.175185] Lustre: 121605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3825.241139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.668830] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3835.016221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3835.025292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3835.150129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3841.367826] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3845.937777] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3846.192454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3851.204519] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3851.459529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3855.840319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.843708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.846359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.852836] Lustre: Skipped 1 previous similar message [ 3855.862231] Lustre: Skipped 1 previous similar message [ 3859.872541] Lustre: server umount lustre-OST0000 complete [ 3866.088444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.095981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.117024] Lustre: Skipped 1 previous similar message [ 3866.146234] Lustre: Skipped 1 previous similar message [ 3875.305461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3875.319406] Lustre: Skipped 3 previous similar messages [ 3878.367141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3878.668474] Lustre: server umount lustre-MDT0000 complete [ 3883.691903] LustreError: 119631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552029 with bad export cookie 14888971864336641267 [ 3883.704314] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3883.708454] LustreError: 119631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3884.148610] Lustre: server umount lustre-MDT0001 complete [ 3892.310466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.864589] LustreError: 123823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.883673] LustreError: 123823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3893.024008] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.341570] LustreError: 123824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.134778] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3902.435618] LustreError: 123823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.559961] LustreError: 123824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.470641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.927597] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3919.848938] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3925.459313] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.150346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.133953] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3943.207402] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3950.003816] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3954.147211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3954.156236] LustreError: Skipped 1 previous similar message [ 3954.162080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.177405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3958.241384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.486908] Lustre: server umount lustre-OST0000 complete [ 3963.365036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.383961] Lustre: Skipped 2 previous similar messages [ 3969.625657] Lustre: server umount lustre-MDT0000 complete [ 3973.337762] LustreError: 124560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552119 with bad export cookie 14888971864336642149 [ 3973.342348] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3973.350844] LustreError: 124560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3973.749688] Lustre: server umount lustre-MDT0001 complete [ 3983.845466] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3986.962963] Key type lgssc unregistered [ 3987.264795] LNet: 127457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3987.293185] LNetError: 127457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3988.329714] LNet: Removed LNI 192.168.203.101@tcp [ 3989.403214] Key type .llcrypt unregistered [ 3989.406890] Key type ._llcrypt unregistered [ 4007.716797] Key type ._llcrypt registered [ 4007.721949] Key type .llcrypt registered [ 4007.833448] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 4022.562297] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4023.706385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4023.733963] alg: No test for adler32 (adler32-zlib) [ 4024.752441] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 4024.996579] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4026.664093] Key type lgssc registered [ 4027.923784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4059.334957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4059.368669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.669353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.741596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.862265] Lustre: lustre-MDT0000: new disk, initializing [ 4060.998670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.024837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.137619] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4075.160033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.314562] Lustre: 130606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4075.404393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4075.409507] Lustre: Skipped 1 previous similar message [ 4075.508791] Lustre: lustre-MDT0001: new disk, initializing [ 4075.636425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.686571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4075.713702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4079.658448] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4088.293506] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4093.686917] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4099.452500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.747976] Lustre: lustre-OST0000: new disk, initializing [ 4099.755834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4099.783981] Lustre: 132577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4099.951232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.423663] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4106.800348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4106.822798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4106.965801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4113.677200] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4117.830077] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4118.131101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4122.834663] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4123.104775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4127.720318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.723552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.724818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4127.751428] Lustre: Skipped 1 previous similar message [ 4131.167456] Lustre: server umount lustre-OST0000 complete [ 4136.930967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4136.948684] LustreError: Skipped 1 previous similar message [ 4136.956107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.970900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.973457] Lustre: Skipped 1 previous similar message [ 4137.958940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.978988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.982365] Lustre: Skipped 1 previous similar message [ 4143.072373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.081160] Lustre: Skipped 1 previous similar message [ 4147.169950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4147.174701] Lustre: Skipped 1 previous similar message [ 4149.215145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4149.556850] Lustre: server umount lustre-MDT0000 complete [ 4153.180532] LustreError: 130601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552299 with bad export cookie 1815936894634495931 [ 4153.198561] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.591435] Lustre: server umount lustre-MDT0001 complete [ 4161.365685] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:18:25 (1787552305) [ 4167.869731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.197392] LustreError: 134997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.206801] LustreError: 134997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4168.273970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.129522] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4173.297700] LustreError: 134998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.403222] LustreError: 134997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.969407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.451905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4186.684841] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4194.166317] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4199.163533] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4202.976806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4202.981174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.992506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.263184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4215.450958] Lustre: server umount lustre-MDT0000 complete [ 4218.345393] LustreError: 134997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.363925] LustreError: 134997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4219.025628] LustreError: 134979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552364 with bad export cookie 1815936894634496813 [ 4219.046247] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4219.051579] LustreError: 134979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4219.372959] Lustre: server umount lustre-MDT0001 complete [ 4226.837641] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:19:30 (1787552370) [ 4233.096180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4248.672199] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4253.791604] LustreError: 137413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 4279.455441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.630510] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4295.998455] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4298.979573] Lustre: server umount lustre-OST0000 complete [ 4308.695608] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:20:52 (1787552452) [ 4315.401619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.721426] LustreError: 138863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.829536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.468266] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4325.347530] LustreError: 138862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.362406] LustreError: 138862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4328.672969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.035099] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4340.820564] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4345.766741] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.506985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.864648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.871832] Lustre: Skipped 1 previous similar message [ 4357.003108] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4363.458587] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4371.429110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.430938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4371.448537] Lustre: Skipped 3 previous similar messages [ 4371.451359] Lustre: Skipped 7 previous similar messages [ 4376.802534] Lustre: server umount lustre-MDT0000 complete [ 4379.955862] LustreError: 138843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552525 with bad export cookie 1815936894634497471 [ 4379.968995] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.974501] LustreError: 138843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4380.267353] Lustre: server umount lustre-MDT0001 complete [ 4393.859730] Lustre: server umount lustre-OST0000 complete [ 4401.473220] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:22:25 (1787552545) [ 4407.264647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.844511] LustreError: 142512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.069202] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4419.910376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.308908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4420.314031] Lustre: Skipped 1 previous similar message [ 4424.196433] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4430.793954] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4436.130781] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4441.874936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.187881] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4452.409823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4454.007404] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4458.088328] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4458.320743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4462.960574] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4463.146804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4465.644427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4465.656263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.674921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4465.682779] Lustre: Skipped 4 previous similar messages [ 4470.857416] Lustre: server umount lustre-OST0000 complete [ 4476.898747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4480.546191] Lustre: server umount lustre-MDT0000 complete [ 4483.045243] LustreError: 144953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.062405] LustreError: 144953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4483.707233] LustreError: 142493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552629 with bad export cookie 1815936894634498549 [ 4483.708092] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.717235] LustreError: 142493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4483.931890] Lustre: server umount lustre-MDT0001 complete [ 4491.869068] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:23:55 (1787552635) [ 4498.844868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.399647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4519.519529] LustreError: 146671:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 4551.009618] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4558.154424] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4564.444669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.754731] LustreError: 146696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.806164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.809600] Lustre: Skipped 2 previous similar messages [ 4564.871334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4568.624131] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4575.722668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.895866] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4587.645779] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4593.348196] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4598.702226] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4599.089067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4604.386042] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4604.650150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4606.945620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4606.952232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.964932] Lustre: Skipped 3 previous similar messages [ 4606.972849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.978053] Lustre: Skipped 5 previous similar messages [ 4612.794570] Lustre: server umount lustre-MDT0000 complete [ 4616.135210] LustreError: 146678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552762 with bad export cookie 1815936894634499571 [ 4616.140402] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.150247] LustreError: 146678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4616.391830] Lustre: server umount lustre-MDT0001 complete [ 4630.090596] Lustre: server umount lustre-OST0000 complete [ 4637.340924] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:26:21 (1787552781) [ 4644.793262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4660.383922] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4665.503497] LustreError: 150869:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 4697.039824] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4704.931690] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4710.766581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.049902] LustreError: 150893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.066626] LustreError: 150893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4711.183931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4715.195262] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4722.076707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.649384] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4735.113978] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4741.108917] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4748.306817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4748.540706] Lustre: lustre-OST0001: new disk, initializing [ 4748.551845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4748.559744] Lustre: 153740:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4755.238605] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4756.554028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4756.568217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4756.646062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4764.198548] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4768.185398] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4768.368965] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4772.445599] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4772.648478] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4777.448713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.451857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4777.465554] Lustre: Skipped 3 previous similar messages [ 4777.481690] Lustre: Skipped 4 previous similar messages [ 4780.577388] Lustre: server umount lustre-OST0000 complete [ 4787.683115] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4789.895153] Lustre: server umount lustre-OST0001 complete [ 4799.630978] Lustre: server umount lustre-MDT0000 complete [ 4803.281140] LustreError: 150877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552949 with bad export cookie 1815936894634500523 [ 4803.287137] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4803.296954] LustreError: 150877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4803.920070] Lustre: server umount lustre-MDT0001 complete [ 4829.998534] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:29:33 (1787552973) [ 4832.785378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4834.783588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4836.896269] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:29:40 (1787552980) [ 4843.672923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.965468] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4844.004726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4844.282877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.293485] Lustre: Skipped 5 previous similar messages [ 4848.866422] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4856.081711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.177349] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4856.198329] Lustre: 157300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4860.293655] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4865.937424] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4870.797782] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4877.148077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4877.381366] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4883.377178] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4886.116104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4889.268406] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4893.490514] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4893.681085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4897.781799] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4897.971428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4901.351467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4901.352684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.365489] LustreError: Skipped 2 previous similar messages [ 4901.383480] Lustre: Skipped 5 previous similar messages [ 4905.918829] Lustre: server umount lustre-OST0000 complete [ 4927.641782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4935.512830] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 4941.351796] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.363389] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4941.369866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.828278] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4950.908150] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4951.331585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4957.632949] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4958.018560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4967.299771] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b4b0927d000.ost_server_uuid 50 [ 4969.298427] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b4b0927d000.ost_server_uuid in IDLE state after 0 sec [ 4969.550545] LustreError: 157305:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.1@tcp) returned error from blocking AST (req@00000000d9201bf2 x1874384304467200 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f16d29e6/0x193380fc03372b05 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.1@tcp remote: 0xbf5a9a3eea31bfbd expref: 5 pid: 160236 timeout: 5069 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4969.591382] LustreError: lustre-MDT0000: A client on nid 192.168.203.1@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4969.596815] LustreError: 157295:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.1@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f16d29e6/0x193380fc03372b05 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.1@tcp remote: 0xbf5a9a3eea31bfbd expref: 6 pid: 160236 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4975.077510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4979.824911] Lustre: server umount lustre-OST0000 complete [ 4989.904615] Lustre: server umount lustre-MDT0000 complete [ 4992.487062] LustreError: 157306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.500145] LustreError: 157306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4992.838074] LustreError: 157290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553138 with bad export cookie 1815936894634501706 [ 4992.848602] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5007.327165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5007.595598] Lustre: server umount lustre-MDT0001 complete [ 5018.162242] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5021.190545] Key type lgssc unregistered [ 5021.543577] LNet: 163422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5021.548753] LNetError: 163422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5021.570440] LNet: Removed LNI 192.168.203.101@tcp [ 5022.403064] Key type .llcrypt unregistered [ 5022.406929] Key type ._llcrypt unregistered [ 5044.089383] Key type ._llcrypt registered [ 5044.096438] Key type .llcrypt registered [ 5044.217907] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5045.389123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5045.473196] alg: No test for adler32 (adler32-zlib) [ 5046.643855] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5046.930727] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5048.631162] Key type lgssc registered [ 5049.927479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5057.341399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5057.367613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.714531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.587778] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5068.266201] LustreError: 164833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.293593] LustreError: 164833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5073.380954] LustreError: 164832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.161615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.784479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.103763] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5088.893571] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5093.366501] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5098.547459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.034542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.164131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5104.149276] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5111.021318] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5117.159088] Lustre: Failing over lustre-MDT0000 [ 5117.555537] Lustre: server umount lustre-MDT0000 complete [ 5118.442273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.451753] LustreError: 164828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.463260] Lustre: Skipped 1 previous similar message [ 5118.477394] LustreError: 164828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5123.559190] LustreError: 164829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.576586] LustreError: 164829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5128.675306] LustreError: 166728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.700059] LustreError: 166728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5129.689529] LustreError: 164814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.1@tcp arrived at 1787553275 with bad export cookie 16749679452877155575 [ 5134.819581] Lustre: 164145:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553264/real 1787553264] req@000000007b90a5f1 x1874385375881728/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787553280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5134.850362] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.251745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.938994] LustreError: 167323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.962546] LustreError: 167323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5145.398243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.460347] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5149.908608] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5150.701689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5150.702429] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5152.351559] Lustre: lustre-MDT0000: Denying connection for new client 163a5bde-3dd6-45d6-bcf8-2792eb9f1234 (at 192.168.203.1@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5160.930810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.946357] Lustre: Skipped 1 previous similar message [ 5160.954386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5166.052973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5166.060540] Lustre: Skipped 2 previous similar messages [ 5166.431866] Lustre: server umount lustre-OST0000 complete [ 5170.259254] LustreError: 168563:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5170.266711] Lustre: 167911:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5170.276731] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5170.298120] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5170.307859] LustreError: 167911:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000054549c2c x1874385375901184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5170.320610] LustreError: 167911:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5170.329301] LustreError: 167911:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5170.337320] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5170.586229] Lustre: server umount lustre-MDT0000 complete [ 5171.181108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.202250] Lustre: Skipped 1 previous similar message [ 5171.209904] LustreError: 164829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.236278] LustreError: 164829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5173.791817] LustreError: 164812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553319 with bad export cookie 16749679452877155673 [ 5173.797781] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5173.813269] LustreError: 164812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5176.115756] Lustre: server umount lustre-MDT0001 complete [ 5186.203554] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5189.086836] Key type lgssc unregistered [ 5189.469653] LNet: 169319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5189.478900] LNetError: 169319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5189.498833] LNet: Removed LNI 192.168.203.101@tcp [ 5190.395618] Key type .llcrypt unregistered [ 5190.399771] Key type ._llcrypt unregistered [ 5211.530919] Key type ._llcrypt registered [ 5211.536567] Key type .llcrypt registered [ 5211.711712] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5212.625437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5212.645937] alg: No test for adler32 (adler32-zlib) [ 5213.667943] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5213.980129] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5215.727223] Key type lgssc registered [ 5217.037718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5224.060875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5224.077684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.524171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.184695] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5235.684889] LustreError: 170720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.705579] LustreError: 170720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5238.562731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.928642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5242.673737] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5249.691328] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5255.104147] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5262.613676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.930518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.012456] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5270.149045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5278.470832] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5284.834793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.858064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.883545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.861155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5285.861612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.862858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.565448] Lustre: server umount lustre-OST0000 complete [ 5295.074997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5295.084255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.106644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.115652] Lustre: Skipped 1 previous similar message [ 5301.056914] Lustre: server umount lustre-MDT0000 complete [ 5301.225313] LustreError: 171448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.252522] LustreError: 171448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5304.924628] LustreError: 170700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553450 with bad export cookie 12205190086041825216 [ 5304.946410] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5305.299903] Lustre: server umount lustre-MDT0001 complete [ 5315.269411] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5317.609610] Key type lgssc unregistered [ 5317.818213] LNet: 174255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5317.824458] LNetError: 174255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5317.841188] LNet: Removed LNI 192.168.203.101@tcp [ 5318.528412] Key type .llcrypt unregistered [ 5318.530733] Key type ._llcrypt unregistered [ 5348.334326] Key type ._llcrypt registered [ 5348.336112] Key type .llcrypt registered [ 5348.475688] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5349.238348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5349.451069] alg: No test for adler32 (adler32-zlib) [ 5350.418666] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5350.630525] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5352.311168] Key type lgssc registered [ 5353.612012] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5360.186873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5360.207348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.583709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.464490] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5371.874950] LustreError: 175963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.895520] LustreError: 175963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5372.501505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.857843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5376.619869] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5383.329935] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5388.196695] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5393.816417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.095029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.802280] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5405.967118] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5406.386932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5416.934505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.940456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5416.948297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5416.962291] Lustre: Skipped 1 previous similar message [ 5422.048459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5422.054193] Lustre: Skipped 1 previous similar message [ 5422.659722] Lustre: server umount lustre-OST0000 complete [ 5427.171643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.189242] Lustre: Skipped 1 previous similar message [ 5427.200463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5432.202547] Lustre: server umount lustre-MDT0000 complete [ 5432.289069] LustreError: 175959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.301404] LustreError: 175959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5435.079836] LustreError: 177803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553581 with bad export cookie 54685761766280407 [ 5435.085787] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5435.089209] LustreError: 177803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5435.340743] Lustre: server umount lustre-MDT0001 complete [ 5444.403946] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5447.022536] Key type lgssc unregistered [ 5447.275995] LNet: 179495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5447.283392] LNetError: 179495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5447.297945] LNet: Removed LNI 192.168.203.101@tcp [ 5447.970260] Key type .llcrypt unregistered [ 5447.972709] Key type ._llcrypt unregistered [ 5465.913660] Key type ._llcrypt registered [ 5465.915745] Key type .llcrypt registered [ 5465.992371] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5466.853873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5466.864728] alg: No test for adler32 (adler32-zlib) [ 5467.947207] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5468.173540] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5469.839204] Key type lgssc registered [ 5470.703219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5480.327666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5480.347124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.761332] Lustre: *** cfs_fail_loc=135, val=0*** [ 5481.894887] LustreError: 180997:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5481.914682] LustreError: 180997:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -2 [ 5481.929892] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5481.929892] [ 5481.935679] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5481.956046] LustreError: 180958:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5481.963762] LustreError: 180958:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5481.970973] LustreError: 180958:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5481.976658] LustreError: 180958:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5482.098837] Lustre: server umount lustre-MDT0000 complete [ 5482.104843] LustreError: 180958:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5488.733117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.287624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.730938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5499.881195] LustreError: 181368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.901925] LustreError: 181368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5502.259082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.518156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5507.032097] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5513.010703] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5517.378408] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5519.842631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.846651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5519.866957] Lustre: Skipped 1 previous similar message [ 5519.880983] Lustre: Skipped 1 previous similar message [ 5524.968431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5524.975465] Lustre: Skipped 1 previous similar message [ 5525.241557] Lustre: server umount lustre-MDT0000 complete [ 5528.006109] LustreError: 181348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553673 with bad export cookie 9217543935256545707 [ 5528.008373] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.016047] LustreError: 181348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5528.275987] Lustre: server umount lustre-MDT0001 complete [ 5537.840626] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5540.630401] Key type lgssc unregistered [ 5540.894862] LNet: 183693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5540.901264] LNetError: 183693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5540.911109] LNet: Removed LNI 192.168.203.101@tcp [ 5541.612194] Key type .llcrypt unregistered [ 5541.614495] Key type ._llcrypt unregistered [ 5563.914139] Key type ._llcrypt registered [ 5563.918266] Key type .llcrypt registered [ 5564.000317] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5564.296559] Key type .llcrypt unregistered [ 5564.298124] Key type ._llcrypt unregistered [ 5581.135917] Key type ._llcrypt registered [ 5581.137370] Key type .llcrypt registered [ 5581.244351] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5582.049824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5582.140838] alg: No test for adler32 (adler32-zlib) [ 5583.257477] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5583.587118] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5585.271347] Key type lgssc registered [ 5586.277413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5594.672314] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5594.707304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.271438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5615.392677] LustreError: 185826:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 5642.079178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.250449] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5654.932122] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5661.528650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.983540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5662.081805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5665.873123] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5667.303191] LustreError: 185852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.321189] LustreError: 185852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5672.420409] LustreError: 185851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.438545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.707939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5676.183027] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5683.363099] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5687.911724] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5689.909724] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5703.138750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5703.172613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.197423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.652495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.712918] Lustre: server umount lustre-OST0000 complete [ 5713.034593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.052691] Lustre: Skipped 1 previous similar message [ 5713.058461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.061086] Lustre: Skipped 1 previous similar message [ 5718.496104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.508133] Lustre: Skipped 2 previous similar messages [ 5719.360260] Lustre: server umount lustre-MDT0000 complete [ 5722.725180] LustreError: 185834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553868 with bad export cookie 1142188754381488954 [ 5722.728599] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.736029] LustreError: 185834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5723.197385] Lustre: server umount lustre-MDT0001 complete [ 5732.974983] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5735.427698] Key type lgssc unregistered [ 5735.697926] LNet: 189890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5735.708407] LNetError: 189890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5735.729736] LNet: Removed LNI 192.168.203.101@tcp [ 5736.527853] Key type .llcrypt unregistered [ 5736.534203] Key type ._llcrypt unregistered [ 5755.351798] Key type ._llcrypt registered [ 5755.354237] Key type .llcrypt registered [ 5755.447695] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5756.343852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5756.446313] alg: No test for adler32 (adler32-zlib) [ 5757.504964] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5757.728482] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5759.375463] Key type lgssc registered [ 5760.359704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5766.610373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5766.632186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.003188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.376649] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5777.387881] LustreError: 191282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.396967] LustreError: 191282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5778.080036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.473586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5782.525314] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5789.398211] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5794.384120] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5800.026985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.370627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.528626] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5812.002485] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5812.668609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5820.168391] Lustre: Failing over lustre-MDT0000 [ 5820.361186] Lustre: server umount lustre-MDT0000 complete [ 5822.660846] LustreError: 191277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.949942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.972587] Lustre: Skipped 2 previous similar messages [ 5827.725904] LustreError: 191277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.752410] LustreError: 191277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5832.856157] LustreError: 191278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.889320] LustreError: 191278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5837.955282] LustreError: 192714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.975080] LustreError: 192714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5838.278745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.303150] Lustre: 190594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553968/real 1787553968] req@00000000c3c50e0f x1874386121416832/t0(0) o400->MGC192.168.203.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787553984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5838.332963] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5848.207344] LustreError: 191277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.235516] LustreError: 191277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5848.549204] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4888051b0e7de872 [ 5848.568097] Lustre: MGC192.168.203.101@tcp: Connection restored to 0@lo (at 0@lo) [ 5848.796779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.853568] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5852.739359] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5853.330981] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5854.209336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5854.234701] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5854.281604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5856.708853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5858.288365] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5862.883969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5862.889880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.899066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.417563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.418526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.891775] Lustre: server umount lustre-OST0000 complete [ 5874.656043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.657330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5874.658559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.658564] Lustre: Skipped 1 previous similar message [ 5874.668421] Lustre: Skipped 1 previous similar message [ 5878.725538] Lustre: server umount lustre-MDT0000 complete [ 5879.783862] LustreError: 191282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.813591] LustreError: 191282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5881.900149] LustreError: 191263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554027 with bad export cookie 5226432981328849010 [ 5881.909756] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.912794] LustreError: 191263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5882.209553] Lustre: server umount lustre-MDT0001 complete [ 5894.728823] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 5897.879455] Key type lgssc unregistered [ 5898.296728] LNet: 196022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5898.301891] LNetError: 196022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5898.315799] LNet: Removed LNI 192.168.203.101@tcp [ 5899.124279] Key type .llcrypt unregistered [ 5899.128368] Key type ._llcrypt unregistered [ 5921.377853] Key type ._llcrypt registered [ 5921.382842] Key type .llcrypt registered [ 5921.483408] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 5922.601824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5922.647786] alg: No test for adler32 (adler32-zlib) [ 5923.638529] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5923.977620] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 5925.703155] Key type lgssc registered [ 5927.054699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5935.475838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5935.518053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.124335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.955057] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5946.348709] LustreError: 197437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.364198] LustreError: 197437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5949.598908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.018667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5953.315447] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5960.453181] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5965.708662] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5972.517226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5972.904922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.169715] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 5981.108599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5986.184641] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5989.668985] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5996.209484] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6008.178192] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6015.969529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6015.982963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.009779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6016.993233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6016.994882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.995297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6020.553655] Lustre: server umount lustre-OST0000 complete [ 6026.721349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6026.730467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.745082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6026.751513] Lustre: Skipped 1 previous similar message [ 6030.227570] Lustre: server umount lustre-MDT0000 complete [ 6032.352373] LustreError: 197434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.371941] LustreError: 197434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6033.187658] LustreError: 197417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554179 with bad export cookie 4834792959082996147 [ 6033.191109] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6033.194410] LustreError: 197417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6033.405524] Lustre: server umount lustre-MDT0001 complete [ 6043.587210] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6045.998425] Key type lgssc unregistered [ 6046.292935] LNet: 201289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6046.308455] LNetError: 201289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6046.338536] LNet: Removed LNI 192.168.203.101@tcp [ 6047.147157] Key type .llcrypt unregistered [ 6047.149115] Key type ._llcrypt unregistered [ 6064.684587] Key type ._llcrypt registered [ 6064.686257] Key type .llcrypt registered [ 6064.839699] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6065.299563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6065.340362] alg: No test for adler32 (adler32-zlib) [ 6066.376955] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6066.629265] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6068.287154] Key type lgssc registered [ 6069.294325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6078.485730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6078.538194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.263760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.485730] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6090.736509] LustreError: 202555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.767561] LustreError: 202555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6092.144659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.502767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.182343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6103.296854] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6107.800848] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6112.980807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6113.332653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.415461] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6124.861949] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6126.645156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6133.862787] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6146.391668] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6155.555812] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6164.275365] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6172.640240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6172.652061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6172.654410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6172.665710] Lustre: Skipped 1 previous similar message [ 6172.674039] Lustre: Skipped 1 previous similar message [ 6176.560101] Lustre: server umount lustre-OST0000 complete [ 6182.882880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.924197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6182.931901] Lustre: Skipped 1 previous similar message [ 6188.002491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6188.025453] Lustre: Skipped 1 previous similar message [ 6188.339693] Lustre: server umount lustre-MDT0000 complete [ 6192.097735] LustreError: 202556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.137428] LustreError: 202556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6192.780836] LustreError: 205232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554338 with bad export cookie 4553845672140811567 [ 6192.786897] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6192.808377] LustreError: 205232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6193.143711] Lustre: server umount lustre-MDT0001 complete [ 6203.900161] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6206.792876] Key type lgssc unregistered [ 6207.096811] LNet: 207580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6207.118048] LNetError: 207580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6207.140278] LNet: Removed LNI 192.168.203.101@tcp [ 6207.923175] Key type .llcrypt unregistered [ 6207.929694] Key type ._llcrypt unregistered [ 6227.585431] Key type ._llcrypt registered [ 6227.587972] Key type .llcrypt registered [ 6227.669983] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6228.339321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6228.391928] alg: No test for adler32 (adler32-zlib) [ 6229.399756] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6229.578167] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6231.279178] Key type lgssc registered [ 6232.344710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6238.834977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6238.858274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.289652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.961333] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6250.469289] LustreError: 208972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.487422] LustreError: 208972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6252.500057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.856917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6256.579146] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6262.874948] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6267.337245] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6273.517969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.937157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.713840] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6285.237622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6286.430058] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6298.595706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6298.605657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.631549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.642191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.645795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6300.661076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.621639] Lustre: server umount lustre-OST0000 complete [ 6309.346289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6309.354350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.370224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6309.375386] Lustre: Skipped 1 previous similar message [ 6314.412566] Lustre: server umount lustre-MDT0000 complete [ 6316.000516] LustreError: 209701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.032230] LustreError: 209701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6317.763953] LustreError: 209708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554463 with bad export cookie 17030610657223647493 [ 6317.764758] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.780303] LustreError: 209708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6318.047336] Lustre: server umount lustre-MDT0001 complete [ 6326.759696] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6329.564902] Key type lgssc unregistered [ 6329.907951] LNet: 212658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6329.919079] LNetError: 212658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6329.941267] LNet: Removed LNI 192.168.203.101@tcp [ 6330.753873] Key type .llcrypt unregistered [ 6330.758917] Key type ._llcrypt unregistered [ 6350.125683] Key type ._llcrypt registered [ 6350.127856] Key type .llcrypt registered [ 6350.261891] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6351.545558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6351.581065] alg: No test for adler32 (adler32-zlib) [ 6352.712852] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6353.033763] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6354.793102] Key type lgssc registered [ 6355.911726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6363.754802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6363.787040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6365.386696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6370.413775] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6375.908436] LustreError: 213926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.931489] LustreError: 213926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6379.363084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.775425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.970320] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6393.001195] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6399.381639] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.243571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.577066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.745195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6413.427256] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6420.779507] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6436.327402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.343307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.375320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6440.417114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.427400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6440.435161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.312668] Lustre: server umount lustre-OST0000 complete [ 6446.560648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6446.572180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.591461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6446.595678] Lustre: Skipped 1 previous similar message [ 6450.659819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.661138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.734358] Lustre: server umount lustre-MDT0000 complete [ 6455.780089] LustreError: 215656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.800989] LustreError: 215656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6456.077382] LustreError: 213906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554601 with bad export cookie 13906446935621598667 [ 6456.099402] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.425982] Lustre: server umount lustre-MDT0001 complete [ 6467.819730] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6470.600889] Key type lgssc unregistered [ 6470.883875] LNet: 217609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6470.893445] LNetError: 217609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.912965] LNet: Removed LNI 192.168.203.101@tcp [ 6471.720137] Key type .llcrypt unregistered [ 6471.721846] Key type ._llcrypt unregistered [ 6490.302970] Key type ._llcrypt registered [ 6490.305379] Key type .llcrypt registered [ 6490.391260] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6491.308124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6491.328582] alg: No test for adler32 (adler32-zlib) [ 6492.424462] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6492.642950] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6494.335289] Key type lgssc registered [ 6495.403457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6503.585675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6503.628568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.082665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.135268] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6514.664732] LustreError: 218896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.690177] LustreError: 218896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6519.281503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.792743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6525.264287] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6532.904302] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6538.973927] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6546.580918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.236334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6549.244066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6553.594762] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6560.748200] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6569.948978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.037070] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6570.264770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6576.553370] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6584.816722] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6586.616295] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6598.820864] Lustre: Permanently deactivating lustre-OST0001 [ 6598.843668] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6608.996920] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6618.202288] Lustre: Permanently reactivating lustre-OST0001 [ 6618.223329] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6618.227637] Lustre: Skipped 2 previous similar messages [ 6626.367398] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.376277] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6626.381573] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6626.390723] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6626.422152] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6631.405092] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.417667] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6631.422716] Lustre: Skipped 2 previous similar messages [ 6633.001879] Lustre: server umount lustre-OST0001 complete [ 6636.523196] LustreError: 222222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.549025] LustreError: 222222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6641.645287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.647543] LustreError: 220719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6641.647930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6641.671549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6641.687418] LustreError: 220719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6641.719309] Lustre: Skipped 2 previous similar messages [ 6642.987920] Lustre: server umount lustre-OST0000 complete [ 6647.782107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6647.789590] LustreError: Skipped 1 previous similar message [ 6647.796094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.817558] Lustre: Skipped 1 previous similar message [ 6647.826917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6651.874910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.877327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6651.889891] Lustre: Skipped 1 previous similar message [ 6656.992558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6657.001186] Lustre: Skipped 1 previous similar message [ 6661.090185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6663.455472] Lustre: server umount lustre-MDT0000 complete [ 6666.209222] LustreError: 218892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.221213] LustreError: 218892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6666.788559] LustreError: 218876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554812 with bad export cookie 17541388987863504678 [ 6666.801000] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.806419] LustreError: 218876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6667.285168] Lustre: server umount lustre-MDT0001 complete [ 6679.375995] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6682.261744] Key type lgssc unregistered [ 6682.633878] LNet: 224019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6682.642404] LNetError: 224019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6682.655749] LNet: Removed LNI 192.168.203.101@tcp [ 6683.631929] Key type .llcrypt unregistered [ 6683.637185] Key type ._llcrypt unregistered [ 6723.000346] Key type ._llcrypt registered [ 6723.005490] Key type .llcrypt registered [ 6723.121687] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6724.307054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6724.336033] alg: No test for adler32 (adler32-zlib) [ 6725.385526] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6725.645267] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6727.367226] Key type lgssc registered [ 6728.587594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6736.048513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6736.060938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.319904] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6737.366418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6737.558047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.292379] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6752.262591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.396567] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6752.452361] Lustre: 226238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6752.731175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6757.317468] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6764.853920] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6771.780576] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6778.464170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.815974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6779.228687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.548759] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 6786.736128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6794.198651] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.867942] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6802.097309] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6809.635039] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6818.482622] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6828.128985] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6846.449993] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6846.455631] Lustre: Skipped 1 previous similar message [ 6879.980655] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6879.987860] Lustre: Skipped 4 previous similar messages [ 6944.914486] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6944.919894] Lustre: Skipped 8 previous similar messages [ 6966.241787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6966.246564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.254865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.269220] Lustre: Skipped 1 previous similar message [ 6971.366404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.375346] Lustre: Skipped 2 previous similar messages [ 6972.109563] Lustre: server umount lustre-OST0000 complete [ 6976.480924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.491332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.494983] Lustre: Skipped 1 previous similar message [ 6981.609185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.623968] Lustre: Skipped 2 previous similar messages [ 6986.721083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6986.734521] Lustre: Skipped 1 previous similar message [ 6989.279117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6989.467186] Lustre: server umount lustre-MDT0000 complete [ 6992.751513] LustreError: 228912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555138 with bad export cookie 16787478181449839303 [ 6992.755237] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6992.768138] LustreError: 228912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6993.090026] Lustre: server umount lustre-MDT0001 complete [ 7003.078548] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7005.524693] Key type lgssc unregistered [ 7005.909779] LNet: 231048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7005.923827] LNetError: 231048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7005.946320] LNet: Removed LNI 192.168.203.101@tcp [ 7006.684780] Key type .llcrypt unregistered [ 7006.687600] Key type ._llcrypt unregistered [ 7028.200756] Key type ._llcrypt registered [ 7028.204591] Key type .llcrypt registered [ 7028.333529] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7029.448802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7029.462579] alg: No test for adler32 (adler32-zlib) [ 7030.540649] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 7030.791685] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7032.487279] Key type lgssc registered [ 7033.439375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7039.911132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7039.930386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.416202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7045.874718] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7051.748216] LustreError: 232454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.765370] LustreError: 232454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7053.577445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.005707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.642167] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7064.271637] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7069.867041] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.167368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.548707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7081.814415] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 7086.803812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7088.967885] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7104.486649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7104.504847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.517423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7105.664933] Lustre: server umount lustre-OST0000 complete [ 7110.624921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7110.638552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.663190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.672971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.674158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.692837] Lustre: Skipped 1 previous similar message [ 7117.794647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7117.798299] Lustre: Skipped 1 previous similar message [ 7118.877420] Lustre: server umount lustre-MDT0000 complete [ 7122.389912] LustreError: 232436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555268 with bad export cookie 2769685294964229956 [ 7122.393364] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7122.410438] LustreError: 232436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7122.785344] Lustre: server umount lustre-MDT0001 complete [ 7133.157301] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7135.628783] Key type lgssc unregistered [ 7135.896062] LNet: 236145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7135.904317] LNetError: 236145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7135.925889] LNet: Removed LNI 192.168.203.101@tcp [ 7136.725237] Key type .llcrypt unregistered [ 7136.727995] Key type ._llcrypt unregistered [ 7159.066717] Key type ._llcrypt registered [ 7159.072348] Key type .llcrypt registered [ 7159.265121] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7160.624875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7160.650084] alg: No test for adler32 (adler32-zlib) [ 7161.678397] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 7161.927276] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7163.623174] Key type lgssc registered [ 7164.801761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7436.717081] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7440.165639] Key type lgssc unregistered [ 7440.563708] LNet: 238172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7440.573214] LNetError: 238172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7440.601410] LNet: Removed LNI 192.168.203.101@tcp [ 7441.536663] Key type .llcrypt unregistered [ 7441.540358] Key type ._llcrypt unregistered [ 7464.869194] Key type ._llcrypt registered [ 7464.871935] Key type .llcrypt registered [ 7464.956815] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 7478.919983] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 7479.927236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7479.947146] alg: No test for adler32 (adler32-zlib) [ 7480.963880] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 7481.192959] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 7482.847168] Key type lgssc registered [ 7483.952848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7510.871554] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7182 sec ========== 03:14:14 (1787555654) [ 7512.286297] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:14:16 (1787555656) === [ 7515.498488] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:14:19 (1787555659) === [ 7550.530379] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 7552.845656] Key type lgssc unregistered [ 7553.093521] LNet: 243408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7553.102599] LNetError: 243408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7553.118723] LNet: Removed LNI 192.168.203.101@tcp [ 7554.001145] Key type .llcrypt unregistered [ 7554.003179] Key type ._llcrypt unregistered