[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 448598351 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011045] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.021219] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026365] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] *** VALIDATE tmpfs *** [ 0.031451] *** VALIDATE proc *** [ 0.033082] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042785] debug: unmapping init [mem 0xffffffffb2a59000-0xffffffffb2a60fff] [ 0.044249] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045733] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053345] rcu: Hierarchical SRCU implementation. [ 0.055449] smp: Bringing up secondary CPUs ... [ 0.056633] x86: Booting SMP configuration: [ 0.057031] .... node #0, CPUs: #1 #2 #3 [ 0.060467] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146303] node 0 deferred pages initialised in 81ms [ 0.150134] devtmpfs: initialized [ 0.151254] x86/mm: Memory block size: 128MB [ 0.153696] gcov: version magic: 0x41383552 [ 0.156410] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163254] pinctrl core: initialized pinctrl subsystem [ 0.165191] [ 0.165611] ************************************************************* [ 0.168011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170011] ** ** [ 0.173011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175011] ** ** [ 0.178012] ** This means that this kernel is built to expose internal ** [ 0.180011] ** IOMMU data structures, which may compromise security on ** [ 0.183015] ** your system. ** [ 0.185011] ** ** [ 0.188014] ** If you see this message and you are not debugging the ** [ 0.190013] ** kernel, report this immediately to your vendor! ** [ 0.193020] ** ** [ 0.195012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198014] ************************************************************* [ 0.201656] NET: Registered protocol family 16 [ 0.203396] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212132] cpuidle: using governor menu [ 0.214164] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217529] PCI: Using configuration type 1 for base access [ 0.219118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236074] cryptd: max_cpu_qlen set to 1000 [ 0.240326] ACPI: Added _OSI(Module Device) [ 0.242012] ACPI: Added _OSI(Processor Device) [ 0.243012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.247652] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254365] ACPI: Interpreter enabled [ 0.256065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257012] ACPI: Using IOAPIC for interrupt routing [ 0.259109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.262330] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273236] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287299] acpiphp: Slot [2] registered [ 0.288131] acpiphp: Slot [5] registered [ 0.290123] acpiphp: Slot [6] registered [ 0.291097] acpiphp: Slot [7] registered [ 0.293148] acpiphp: Slot [8] registered [ 0.295123] acpiphp: Slot [9] registered [ 0.296112] acpiphp: Slot [10] registered [ 0.298125] acpiphp: Slot [3] registered [ 0.299162] acpiphp: Slot [4] registered [ 0.301099] acpiphp: Slot [11] registered [ 0.303091] acpiphp: Slot [12] registered [ 0.304059] acpiphp: Slot [13] registered [ 0.305094] acpiphp: Slot [14] registered [ 0.306079] acpiphp: Slot [15] registered [ 0.308097] acpiphp: Slot [16] registered [ 0.310104] acpiphp: Slot [17] registered [ 0.311097] acpiphp: Slot [18] registered [ 0.313225] acpiphp: Slot [19] registered [ 0.314099] acpiphp: Slot [20] registered [ 0.315096] acpiphp: Slot [21] registered [ 0.317107] acpiphp: Slot [22] registered [ 0.319122] acpiphp: Slot [23] registered [ 0.320209] acpiphp: Slot [24] registered [ 0.322098] acpiphp: Slot [25] registered [ 0.324109] acpiphp: Slot [26] registered [ 0.325106] acpiphp: Slot [27] registered [ 0.327106] acpiphp: Slot [28] registered [ 0.328086] acpiphp: Slot [29] registered [ 0.329090] acpiphp: Slot [30] registered [ 0.330078] acpiphp: Slot [31] registered [ 0.331052] PCI host bridge to bus 0000:00 [ 0.333018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350913] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354225] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379796] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383033] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389989] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415912] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.421051] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467113] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502807] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.507015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.512013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.526017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.537059] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.543015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.550014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.573017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.615015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.625322] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.630013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.661021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.675075] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.679457] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.682450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.685598] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.688300] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693039] iommu: Default domain type: Passthrough [ 0.695426] SCSI subsystem initialized [ 0.697130] ACPI: bus type USB registered [ 0.698099] usbcore: registered new interface driver usbfs [ 0.700066] usbcore: registered new interface driver hub [ 0.701078] usbcore: registered new device driver usb [ 0.703133] pps_core: LinuxPPS API ver. 1 registered [ 0.704009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.706067] PTP clock support registered [ 0.708155] EDAC MC: Ver: 3.0.0 [ 0.710051] PCI: Using ACPI for IRQ routing [ 0.711668] NetLabel: Initializing [ 0.713018] NetLabel: domain hash size = 128 [ 0.715024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.719112] NetLabel: unlabeled traffic allowed by default [ 0.721150] vgaarb: loaded [ 0.723259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.725013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.731005] clocksource: Switched to clocksource kvm-clock [ 0.836880] VFS: Disk quotas dquot_6.6.0 [ 0.838864] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.841845] *** VALIDATE ramfs *** [ 0.843324] *** VALIDATE hugetlbfs *** [ 0.845252] pnp: PnP ACPI init [ 0.848189] pnp: PnP ACPI: found 6 devices [ 0.868707] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.871848] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.874851] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.876905] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.880829] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.883616] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.886515] NET: Registered protocol family 2 [ 0.889286] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.894642] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.898032] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.904115] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.908329] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912118] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.915542] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.918635] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923350] NET: Registered protocol family 1 [ 0.926207] RPC: Registered named UNIX socket transport module. [ 0.928613] RPC: Registered udp transport module. [ 0.930555] RPC: Registered tcp transport module. [ 0.932503] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.935104] NET: Registered protocol family 44 [ 0.936889] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.939415] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.941497] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943578] PCI: CLS 0 bytes, default 64 [ 0.945986] Unpacking initramfs... [ 2.350425] debug: unmapping init [mem 0xffff8e89bcc54000-0xffff8e89bffbffff] [ 2.354743] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.357214] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360722] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.863275] Initialise system trusted keyrings [ 2.865777] Key type blacklist registered [ 2.868474] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.879484] zbud: loaded [ 2.883128] *** VALIDATE nfs *** [ 2.884698] *** VALIDATE nfs4 *** [ 2.886607] pstore: using deflate compression [ 2.890868] Platform Keyring initialized [ 2.994047] NET: Registered protocol family 38 [ 2.996070] Key type asymmetric registered [ 2.997694] Asymmetric key parser 'x509' registered [ 2.999906] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.003843] io scheduler mq-deadline registered [ 3.005674] io scheduler kyber registered [ 3.007767] io scheduler bfq registered [ 3.010543] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.014533] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.018129] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.021687] ACPI: Power Button [PWRF] [ 3.027869] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.036517] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.047656] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.053863] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.066787] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.095069] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.123145] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.128194] Non-volatile memory driver v1.3 [ 3.129962] Linux agpgart interface v0.103 [ 3.164405] virtio_blk virtio1: [vda] 134784 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.168032] vda: detected capacity change from 0 to 69009408 [ 3.185196] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.187617] vdb: detected capacity change from 0 to 1073741824 [ 3.201782] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205580] vdc: detected capacity change from 0 to 2621440000 [ 3.225660] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228723] vdd: detected capacity change from 0 to 2621440000 [ 3.244511] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.247977] vde: detected capacity change from 0 to 4294967296 [ 3.267666] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.270917] vdf: detected capacity change from 0 to 4294967296 [ 3.282538] libphy: Fixed MDIO Bus: probed [ 3.289812] usbcore: registered new interface driver usbserial_generic [ 3.293428] usbserial: USB Serial support registered for generic [ 3.295956] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.300623] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302836] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.305569] mousedev: PS/2 mouse device common for all mice [ 3.309268] rtc_cmos 00:05: RTC can wake from S4 [ 3.311916] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.313540] rtc_cmos 00:05: registered as rtc0 [ 3.318270] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.321526] intel_pstate: CPU model not supported [ 3.324795] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.327614] hid: raw HID events driver (C) Jiri Kosina [ 3.331992] usbcore: registered new interface driver usbhid [ 3.334513] usbhid: USB HID core driver [ 3.335214] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.336166] drop_monitor: Initializing network drop monitor service [ 3.342540] Initializing XFRM netlink socket [ 3.345249] NET: Registered protocol family 10 [ 3.349124] Segment Routing with IPv6 [ 3.351067] NET: Registered protocol family 17 [ 3.353360] mpls_gso: MPLS GSO support [ 3.359553] RAS: Correctable Errors collector initialized. [ 3.362553] AVX version of gcm_enc/dec engaged. [ 3.364630] AES CTR mode by8 optimization enabled [ 3.449984] sched_clock: Marking stable (3449959276, 0)->(4350963873, -901004597) [ 3.453077] registered taskstats version 1 [ 3.455609] Loading compiled-in X.509 certificates [ 3.457557] zswap: loaded using pool lzo/zbud [ 3.483109] Key type big_key registered [ 3.495913] Key type encrypted registered [ 3.497691] ima: No TPM chip found, activating TPM-bypass! [ 3.500365] ima: Allocated hash algorithm: sha1 [ 3.502561] ima: No architecture policies found [ 3.505088] evm: Initialising EVM extended attributes: [ 3.507462] evm: security.selinux [ 3.508979] evm: security.ima [ 3.510383] evm: security.capability [ 3.512316] evm: HMAC attrs: 0x1 [ 3.515257] rtc_cmos 00:05: setting system clock to 2026-05-16 09:01:43 UTC (1778922103) [ 3.522963] debug: unmapping init [mem 0xffffffffb3a03000-0xffffffffb3bfffff] [ 3.526364] debug: unmapping init [mem 0xffffffffb2782000-0xffffffffb2a58fff] [ 3.537443] Write protecting the kernel read-only data: 28672k [ 3.541081] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.544151] debug: unmapping init [mem 0xffffffffb1714000-0xffffffffb17fffff] [ 3.581779] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.590846] systemd[1]: Detected virtualization kvm. [ 3.593131] systemd[1]: Detected architecture x86-64. [ 3.595352] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.624903] systemd[1]: No hostname configured. [ 3.627147] systemd[1]: Set hostname to . [ 3.629750] random: systemd: uninitialized urandom read (16 bytes read) [ 3.632729] systemd[1]: Initializing machine ID from random generator. [ 3.694857] random: ln: uninitialized urandom read (6 bytes read) [ 3.778166] random: systemd: uninitialized urandom read (16 bytes read) [ 3.781445] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.787808] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.794645] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.475956] device-mapper: uevent: version 1.0.3 [ 4.478192] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System...[ 5.020843] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.213669] virtio_net virtio0 ens2: renamed from eth0 [ 5.252329] scsi host0: ata_piix [ 5.312734] scsi host1: ata_piix [ 5.314308] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.317118] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.087458] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.023493] random: crng init done [ 10.025075] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.440926] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.527396] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.788367] SELinux: Disabled at runtime. [ 11.847097] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.855697] systemd[1]: Detected virtualization kvm. [ 11.857632] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.325135] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.327886] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.332648] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.336337] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.339065] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.346341] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.349528] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.492698] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.694514] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.968450] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.019602] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.146789] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.155694] EDAC sbridge: Ver: 1.1.2 [ 14.741396] Key type dns_resolver registered [ 15.041103] NFS: Registering the id_resolver key type [ 15.043253] Key type id_resolver registered [ 15.044809] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg330-server login: [ 42.134181] hrtimer: interrupt took 5428461 ns [ 61.639570] libcfs: loading out-of-tree module taints kernel. [ 61.675921] Key type ._llcrypt registered [ 61.680433] Key type .llcrypt registered [ 61.775562] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 80.608900] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 82.042790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.051227] alg: No test for adler32 (adler32-zlib) [ 83.596590] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 84.569467] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 86.393324] Key type lgssc registered [ 88.362440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.289176] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.717458] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 157.262726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.334316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.544855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.606940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.712477] Lustre: lustre-MDT0000: new disk, initializing [ 158.839042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.862994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.445657] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.205586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.280792] Lustre: 6503:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.302326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.306367] Lustre: Skipped 1 previous similar message [ 177.383549] Lustre: lustre-MDT0001: new disk, initializing [ 177.457189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.492447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.504184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.819528] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.020209] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.873898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.140443] Lustre: lustre-OST0000: new disk, initializing [ 196.142552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.147018] Lustre: 8408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.207924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.262135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.269659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.339570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 202.008850] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.815965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.888504] Lustre: lustre-OST0001: new disk, initializing [ 217.892729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 217.898324] Lustre: 9464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.945128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.644916] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.884089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 227.892287] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 227.963095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.840933] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.382969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.724435] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.455089] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing yml_node [ 257.189645] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 259.919184] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 262.580767] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 264.183755] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat May 16 05:06:02 EDT 2026 [ 279.537694] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 281.148440] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 299.489919] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.493558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.509916] Lustre: Skipped 1 previous similar message [ 299.517069] Lustre: Skipped 3 previous similar messages [ 301.977892] Lustre: server umount lustre-MDT0000 complete [ 305.539307] LustreError: 8407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778922405 with bad export cookie 3556197345893792157 [ 305.545393] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.549194] LustreError: 8407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.738437] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.745951] LustreError: 6510:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.746544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.784565] Lustre: Skipped 3 previous similar messages [ 309.829038] LustreError: 6510:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 310.754199] LustreError: 9474:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.754273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.815154] Lustre: Skipped 2 previous similar messages [ 311.961687] Lustre: server umount lustre-MDT0001 complete [ 315.932758] Lustre: server umount lustre-OST0000 complete [ 319.702149] Lustre: server umount lustre-OST0001 complete [ 325.338940] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 333.712011] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 364.464680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.730505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.756957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.823500] Lustre: lustre-MDT0000: new disk, initializing [ 364.916552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.935572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.587956] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 378.195848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.295875] Lustre: 14979:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.390981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.395754] Lustre: Skipped 1 previous similar message [ 378.474947] Lustre: lustre-MDT0001: new disk, initializing [ 378.580351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.623858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 378.642679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 382.748285] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 390.150949] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 393.420977] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 400.383851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 400.762975] Lustre: lustre-OST0000: new disk, initializing [ 400.769389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 400.778488] Lustre: 16721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 400.877066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.498112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 402.508345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 402.642730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 406.231963] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 413.246594] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 418.879506] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 419.141273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 424.143968] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 424.352733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 428.531837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.536736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.549776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 428.583405] Lustre: Skipped 1 previous similar message [ 433.650173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.663833] Lustre: Skipped 1 previous similar message [ 441.311118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 441.510656] Lustre: server umount lustre-OST0000 complete [ 445.411233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 445.415261] LustreError: Skipped 1 previous similar message [ 445.417745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.428064] Lustre: Skipped 1 previous similar message [ 445.434216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.437081] Lustre: Skipped 2 previous similar messages [ 450.661761] Lustre: server umount lustre-MDT0000 complete [ 454.097705] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778922554 with bad export cookie 3556197345893793690 [ 454.099361] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.108639] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 454.393194] Lustre: server umount lustre-MDT0001 complete [ 458.135291] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:09:16 (1778922556) [ 464.014616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.424156] LustreError: 18839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.438819] LustreError: 18839:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 464.496624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.676671] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 469.984988] LustreError: 18840:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.080510] LustreError: 18839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.493698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.332379] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 483.959661] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 488.230213] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.443532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.844866] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 507.717176] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 520.672775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 520.679694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.689621] Lustre: Skipped 1 previous similar message [ 520.700476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.706127] Lustre: Skipped 2 previous similar messages [ 526.811779] Lustre: server umount lustre-OST0000 complete [ 530.912553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.917815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.925970] Lustre: Skipped 1 previous similar message [ 536.510667] Lustre: server umount lustre-MDT0000 complete [ 539.986254] LustreError: 18821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778922639 with bad export cookie 3556197345893794572 [ 539.987483] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.993587] LustreError: 18821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 540.318303] Lustre: server umount lustre-MDT0001 complete [ 549.172571] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 551.609874] Key type lgssc unregistered [ 551.885248] LNet: 22035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.898680] LNetError: 22035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.914825] LNet: Removed LNI 192.168.203.130@tcp [ 552.637520] Key type .llcrypt unregistered [ 552.640600] Key type ._llcrypt unregistered [ 575.358693] Key type ._llcrypt registered [ 575.362661] Key type .llcrypt registered [ 575.459313] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 576.388610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 576.444632] alg: No test for adler32 (adler32-zlib) [ 577.524794] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 577.794238] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 579.496953] Key type lgssc registered [ 580.527443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 587.817893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 587.840588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.302863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.079924] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 599.522180] LustreError: 23432:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.549392] LustreError: 23432:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 601.803029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.107651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 605.778892] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 610.419154] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 614.387781] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 621.282359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 621.763277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.647861] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 634.204238] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 635.081860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 650.720331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 650.721632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.729993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 650.756712] Lustre: Skipped 1 previous similar message [ 651.825652] Lustre: server umount lustre-OST0000 complete [ 655.840718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.849728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 655.860407] Lustre: Skipped 1 previous similar message [ 655.870342] Lustre: Skipped 2 previous similar messages [ 660.969519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 660.974712] Lustre: Skipped 1 previous similar message [ 661.662290] Lustre: server umount lustre-MDT0000 complete [ 665.035897] LustreError: 23411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778922765 with bad export cookie 16988012252489783325 [ 665.041189] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.051456] LustreError: 23411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 665.354106] Lustre: server umount lustre-MDT0001 complete [ 674.361708] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 676.618578] Key type lgssc unregistered [ 676.861287] LNet: 27049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.872527] LNetError: 27049:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.888684] LNet: Removed LNI 192.168.203.130@tcp [ 677.496752] Key type .llcrypt unregistered [ 677.499747] Key type ._llcrypt unregistered [ 697.549472] Key type ._llcrypt registered [ 697.551953] Key type .llcrypt registered [ 697.656943] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 698.578661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 698.630677] alg: No test for adler32 (adler32-zlib) [ 699.632662] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 699.851758] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 701.567228] Key type lgssc registered [ 702.677980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.014239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.033632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.453547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.202294] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 721.888123] LustreError: 28424:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.912659] LustreError: 28424:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 723.495440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.991910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 727.981788] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 733.696711] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 737.351492] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 750.077107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.436434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.594466] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 756.598502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 761.483713] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 774.115780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.116199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.123089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.134811] Lustre: Skipped 1 previous similar message [ 779.209492] Lustre: server umount lustre-OST0000 complete [ 784.357747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.361128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.373074] Lustre: Skipped 2 previous similar messages [ 784.386297] Lustre: Skipped 1 previous similar message [ 788.795758] Lustre: server umount lustre-MDT0000 complete [ 789.477329] LustreError: 28425:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 789.495441] LustreError: 28425:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 792.002802] LustreError: 28406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778922891 with bad export cookie 12723082807612987037 [ 792.004551] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 792.015236] LustreError: 28406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 792.285174] Lustre: server umount lustre-MDT0001 complete [ 801.401370] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 804.212163] Key type lgssc unregistered [ 804.434691] LNet: 31947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.443433] LNetError: 31947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.457609] LNet: Removed LNI 192.168.203.130@tcp [ 805.173983] Key type .llcrypt unregistered [ 805.178057] Key type ._llcrypt unregistered [ 824.402943] Key type ._llcrypt registered [ 824.405140] Key type .llcrypt registered [ 824.482621] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 825.251097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.271799] alg: No test for adler32 (adler32-zlib) [ 826.291242] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 826.435663] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 828.103240] Key type lgssc registered [ 829.170921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 836.403869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 836.431119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.094177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.496379] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 848.354245] LustreError: 33323:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.372147] LustreError: 33323:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 852.199761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.679376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.077268] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 862.745356] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 866.689436] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 873.346508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.990800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.295176] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 883.258119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 887.193388] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 897.505265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.515786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.539674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.542667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.543414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.543558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 903.505980] Lustre: server umount lustre-OST0000 complete [ 908.769943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.771360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.789513] Lustre: Skipped 1 previous similar message [ 908.801332] Lustre: Skipped 2 previous similar messages [ 913.894874] LustreError: 33319:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.902762] LustreError: 33319:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 913.937352] Lustre: server umount lustre-MDT0000 complete [ 918.287752] LustreError: 33304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923018 with bad export cookie 10236261311245226878 [ 918.299606] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.305813] LustreError: 33304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 918.910101] Lustre: server umount lustre-MDT0001 complete [ 931.286947] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 934.229911] Key type lgssc unregistered [ 934.530081] LNet: 36501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.543196] LNetError: 36501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.567411] LNet: Removed LNI 192.168.203.130@tcp [ 935.539690] Key type .llcrypt unregistered [ 935.543438] Key type ._llcrypt unregistered [ 961.327598] Key type ._llcrypt registered [ 961.330909] Key type .llcrypt registered [ 961.445915] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 962.657774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 962.670680] alg: No test for adler32 (adler32-zlib) [ 963.702493] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 963.989902] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 965.671220] Key type lgssc registered [ 967.236139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.367988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.391266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.314814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.728633] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 990.697864] LustreError: 37899:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.731160] LustreError: 37899:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 994.532034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.972885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 999.907343] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1006.798212] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.693623] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1019.695762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.811173] LustreError: 39493:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.049718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.227228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1026.797384] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1033.749830] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1041.385470] Lustre: lustre-OST0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1045.992953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.995329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1046.016316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.024795] LustreError: Skipped 1 previous similar message [ 1051.104467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.111258] Lustre: Skipped 3 previous similar messages [ 1055.199127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1055.368709] Lustre: server umount lustre-OST0000 complete [ 1081.825878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.827289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1081.832976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.832988] Lustre: Skipped 1 previous similar message [ 1081.841268] Lustre: Skipped 2 previous similar messages [ 1083.679684] Lustre: server umount lustre-MDT0000 complete [ 1086.948035] LustreError: 37895:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.965217] LustreError: 37895:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1088.697208] LustreError: 37879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923188 with bad export cookie 10082618308811976069 [ 1088.705752] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.709891] LustreError: 37879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.170654] Lustre: server umount lustre-MDT0001 complete [ 1107.616613] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1110.604195] Key type lgssc unregistered [ 1110.943526] LNet: 41256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.955834] LNetError: 41256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.984996] LNet: Removed LNI 192.168.203.130@tcp [ 1112.014464] Key type .llcrypt unregistered [ 1112.017616] Key type ._llcrypt unregistered [ 1137.965097] Key type ._llcrypt registered [ 1137.967751] Key type .llcrypt registered [ 1138.067506] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1139.132035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.154599] alg: No test for adler32 (adler32-zlib) [ 1140.338764] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1140.640259] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1142.375194] Key type lgssc registered [ 1143.769790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1152.716572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1152.753072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.667043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.502466] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1164.770453] LustreError: 42656:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.805333] LustreError: 42656:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1169.891192] LustreError: 42655:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.383938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.783204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1175.870916] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1182.605644] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1187.111987] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1193.999249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1194.408702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.533757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1200.924608] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1206.849524] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1212.857995] Lustre: server umount lustre-MDT0000 complete [ 1213.931817] LustreError: 42652:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.946281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.967737] Lustre: Skipped 1 previous similar message [ 1216.939468] LustreError: 42635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923316 with bad export cookie 6451054711036701712 [ 1216.947046] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.949204] LustreError: 42635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1219.013726] LustreError: 44807:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.037806] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 1219.037884] LustreError: 44807:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1220.065988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.087818] Lustre: Skipped 1 previous similar message [ 1220.102724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1224.135870] LustreError: 44758:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.137573] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 1224.180914] LustreError: 44758:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1224.198505] Lustre: Skipped 1 previous similar message [ 1226.212870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1231.327135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1231.333902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1231.353045] Lustre: Skipped 1 previous similar message [ 1231.549931] Lustre: server umount lustre-MDT0001 complete [ 1252.573953] Lustre: server umount lustre-OST0000 complete [ 1269.464942] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1272.206246] Key type lgssc unregistered [ 1272.486739] LNet: 46163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.500246] LNetError: 46163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.525879] LNet: Removed LNI 192.168.203.130@tcp [ 1273.262516] Key type .llcrypt unregistered [ 1273.265995] Key type ._llcrypt unregistered [ 1298.636894] Key type ._llcrypt registered [ 1298.646951] Key type .llcrypt registered [ 1298.774478] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1299.808622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.955730] alg: No test for adler32 (adler32-zlib) [ 1301.171438] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1301.437514] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1303.127231] Key type lgssc registered [ 1304.344682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1311.789486] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1311.805926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.327712] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1332.447586] LustreError: 47534:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 1359.135191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.877882] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1372.636879] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1537.129463] Lustre: server umount lustre-OST0000 complete [ 1553.815540] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1556.749986] Key type lgssc unregistered [ 1557.067302] LNet: 49075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.076717] LNetError: 49075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.121452] LNet: Removed LNI 192.168.203.130@tcp [ 1559.008105] Key type .llcrypt unregistered [ 1559.010335] Key type ._llcrypt unregistered [ 1584.717829] Key type ._llcrypt registered [ 1584.722730] Key type .llcrypt registered [ 1584.861407] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1585.954945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1585.979770] alg: No test for adler32 (adler32-zlib) [ 1587.215701] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1587.418068] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1589.127255] Key type lgssc registered [ 1590.466378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1600.100640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1600.131289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.662302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.967964] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1611.236414] LustreError: 50474:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.270255] LustreError: 50474:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1614.931879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.232804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1619.440684] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1625.143497] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1628.993148] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1636.400686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1636.903451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.428676] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1646.133128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1648.485854] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1651.168497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1651.171531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.187071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1651.202504] Lustre: Skipped 1 previous similar message [ 1651.220906] Lustre: Skipped 1 previous similar message [ 1653.164506] Lustre: server umount lustre-OST0000 complete [ 1661.410150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.417239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.428634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.440735] Lustre: Skipped 1 previous similar message [ 1664.310685] Lustre: server umount lustre-MDT0000 complete [ 1666.531296] LustreError: 50469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.551959] LustreError: 50469:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1668.400409] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923768 with bad export cookie 5949372156183328021 [ 1668.422578] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.426867] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1668.876117] Lustre: server umount lustre-MDT0001 complete [ 1680.834456] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1683.794574] Key type lgssc unregistered [ 1684.103851] LNet: 53649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.119357] LNetError: 53649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.157699] LNet: Removed LNI 192.168.203.130@tcp [ 1685.991583] Key type .llcrypt unregistered [ 1685.997244] Key type ._llcrypt unregistered [ 1709.605161] Key type ._llcrypt registered [ 1709.607220] Key type .llcrypt registered [ 1709.699189] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1710.895766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.950133] alg: No test for adler32 (adler32-zlib) [ 1712.005205] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1712.244489] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1713.911242] Key type lgssc registered [ 1714.949713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1722.148045] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1722.173855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.759331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1742.880458] LustreError: 55017:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 1769.695184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.550813] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1780.302645] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.530868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.044792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.109716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1791.741581] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1792.481978] LustreError: 55043:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.509251] LustreError: 55043:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1797.603561] LustreError: 55042:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.428827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.921985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1805.169275] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1811.424726] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1815.900557] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1818.088794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.097539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.119112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.646720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.646848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.949858] Lustre: server umount lustre-OST0000 complete [ 1847.265020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1847.273125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.297282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.304979] Lustre: Skipped 1 previous similar message [ 1850.336658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.344748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.406557] Lustre: server umount lustre-MDT0000 complete [ 1855.457756] LustreError: 55940:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.482245] LustreError: 55940:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1858.109922] LustreError: 55023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923958 with bad export cookie 11905247531339514104 [ 1858.111038] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.138279] LustreError: 55023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1864.783531] Lustre: server umount lustre-MDT0001 complete [ 1877.322417] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1880.682398] Key type lgssc unregistered [ 1880.901625] LNet: 58400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.912138] LNetError: 58400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.931523] LNet: Removed LNI 192.168.203.130@tcp [ 1881.770110] Key type .llcrypt unregistered [ 1881.772397] Key type ._llcrypt unregistered [ 1906.522981] Key type ._llcrypt registered [ 1906.529583] Key type .llcrypt registered [ 1906.684792] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1907.953875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.193677] alg: No test for adler32 (adler32-zlib) [ 1909.291839] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1909.592357] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1911.431644] Key type lgssc registered [ 1912.643032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1921.419238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.448637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.079128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.068412] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1933.292622] LustreError: 59796:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.313693] LustreError: 59796:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1938.197904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.405423] LustreError: 59797:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.659931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1943.685834] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1950.036443] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1954.503373] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1962.129384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.607421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.754614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1968.881808] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1975.405745] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1994.723079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.723736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.742124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.746645] Lustre: Skipped 1 previous similar message [ 1994.775204] LustreError: Skipped 1 previous similar message [ 1999.494989] Lustre: server umount lustre-OST0000 complete [ 2004.961855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.978382] Lustre: Skipped 1 previous similar message [ 2004.980328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.005471] Lustre: Skipped 2 previous similar messages [ 2010.054492] Lustre: server umount lustre-MDT0000 complete [ 2010.091107] LustreError: 59796:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.181301] LustreError: 59796:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.198775] LustreError: 59796:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2014.282933] LustreError: 59778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924114 with bad export cookie 6032109260904996988 [ 2014.330211] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.667779] Lustre: server umount lustre-MDT0001 complete [ 2027.250666] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2030.787440] Key type lgssc unregistered [ 2031.205777] LNet: 62996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.217328] LNetError: 62996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.242605] LNet: Removed LNI 192.168.203.130@tcp [ 2032.050365] Key type .llcrypt unregistered [ 2032.052949] Key type ._llcrypt unregistered [ 2063.471299] Key type ._llcrypt registered [ 2063.474918] Key type .llcrypt registered [ 2063.593188] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2064.730404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.807028] alg: No test for adler32 (adler32-zlib) [ 2066.107515] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2066.468707] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2068.207260] Key type lgssc registered [ 2069.450569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2079.589634] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 05:36:17 (1778924177) [ 2086.371235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2086.403092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.067144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.468853] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2097.635743] LustreError: 64685:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.651648] LustreError: 64685:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2102.757239] LustreError: 64686:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.417537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.979277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.984774] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2116.606485] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2122.182988] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2129.647903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2129.912483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.986958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2137.208310] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2143.388182] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2152.528254] Lustre: Failing over lustre-MDT0000 [ 2152.941562] Lustre: server umount lustre-MDT0000 complete [ 2153.440849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.445630] LustreError: 64685:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.477176] Lustre: Skipped 2 previous similar messages [ 2153.500684] LustreError: 64685:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2155.510288] LustreError: 64680:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.585286] LustreError: 64680:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.623422] LustreError: 64680:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2163.519330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.691758] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.708097] LustreError: 63750:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a919b33c x1865337186317696/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.854676] Lustre: *** cfs_fail_loc=135, val=0*** [ 2163.954981] LustreError: 67433:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2163.961850] LustreError: 67433:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.130@tcp: cfg command failed: rc = -2 [ 2163.970797] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2163.970797] [ 2163.986709] LustreError: MGC192.168.203.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2164.002385] LustreError: 67421:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2164.012717] LustreError: 67421:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2164.028344] LustreError: 67421:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2164.031212] LustreError: 67421:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2164.053656] Lustre: server umount lustre-MDT0000 complete [ 2164.056857] LustreError: 67421:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2168.806349] LustreError: 64685:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.833359] LustreError: 64685:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 2171.695175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.887744] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.913681] LustreError: 64677:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2172.155072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.206424] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2175.944064] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2177.238043] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2177.516617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2177.600495] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2177.646165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2187.745492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.763379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2192.622527] Lustre: server umount lustre-OST0000 complete [ 2197.312920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.345796] Lustre: Skipped 1 previous similar message [ 2197.380333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.393325] Lustre: Skipped 2 previous similar messages [ 2202.607806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.616772] Lustre: Skipped 1 previous similar message [ 2203.765865] Lustre: server umount lustre-MDT0000 complete [ 2207.717551] LustreError: 65396:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.738317] LustreError: 65396:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2208.151787] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924308 with bad export cookie 5513561370914971744 [ 2208.160234] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2208.169091] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2208.638628] Lustre: server umount lustre-MDT0001 complete [ 2221.186520] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2224.247330] Key type lgssc unregistered [ 2224.544953] LNet: 69181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.568643] LNetError: 69181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.603260] LNet: Removed LNI 192.168.203.130@tcp [ 2225.561227] Key type .llcrypt unregistered [ 2225.564773] Key type ._llcrypt unregistered [ 2251.347650] Key type ._llcrypt registered [ 2251.349770] Key type .llcrypt registered [ 2251.464121] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2252.730032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.761578] alg: No test for adler32 (adler32-zlib) [ 2253.856711] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2254.116189] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2255.815209] Key type lgssc registered [ 2256.855483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2264.865988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2264.890910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.511912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2270.906615] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2276.835227] LustreError: 70578:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.852753] LustreError: 70578:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2279.317995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.729915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.580621] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2289.477399] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2293.539942] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2300.639475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2300.985901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.066144] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2311.263320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2314.532223] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2321.355108] Lustre: Failing over lustre-MDT0000 [ 2321.385049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.399919] Lustre: Skipped 2 previous similar messages [ 2321.410710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.417055] Lustre: Skipped 1 previous similar message [ 2321.645152] Lustre: server umount lustre-MDT0000 complete [ 2322.397863] LustreError: 70574:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.501061] LustreError: 71290:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.617815] LustreError: 70574:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.642933] LustreError: 70574:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 2334.783773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.125037] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.528719] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2335.734616] LustreError: 73326:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2335.745242] LustreError: 73326:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.130@tcp: cfg command failed: rc = -9 [ 2335.787614] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2335.787614] [ 2335.817630] LustreError: MGC192.168.203.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2335.860317] LustreError: 73313:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2335.879754] LustreError: 73313:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2335.893271] LustreError: 73313:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2335.908331] LustreError: 73313:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2335.969849] Lustre: server umount lustre-MDT0000 complete [ 2335.978738] LustreError: 73313:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2336.737797] LustreError: 70578:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.778436] LustreError: 70578:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2346.744461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.984533] LustreError: 70578:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.025273] LustreError: 70578:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 2347.208832] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.241360] LustreError: 70570:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2347.927487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.044540] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2349.001852] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2352.053600] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2353.155652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2353.187904] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2353.230834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2360.292958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.301511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.322186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.328443] Lustre: Skipped 1 previous similar message [ 2363.365682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.368867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2363.421053] Lustre: Skipped 1 previous similar message [ 2366.021076] Lustre: server umount lustre-OST0000 complete [ 2373.601655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.602208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2373.632807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.414274] Lustre: server umount lustre-MDT0000 complete [ 2378.721668] LustreError: 70574:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.742346] LustreError: 70574:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2380.470240] LustreError: 70559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924480 with bad export cookie 5159903124722399786 [ 2380.479798] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.782130] Lustre: server umount lustre-MDT0001 complete [ 2392.222484] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2395.255430] Key type lgssc unregistered [ 2395.605826] LNet: 75075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.623555] LNetError: 75075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.648578] LNet: Removed LNI 192.168.203.130@tcp [ 2396.584192] Key type .llcrypt unregistered [ 2396.586737] Key type ._llcrypt unregistered [ 2410.842261] Key type ._llcrypt registered [ 2410.844469] Key type .llcrypt registered [ 2410.940724] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2411.261646] Key type .llcrypt unregistered [ 2411.267214] Key type ._llcrypt unregistered [ 2435.437111] Key type ._llcrypt registered [ 2435.440723] Key type .llcrypt registered [ 2435.544981] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2436.784133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2436.826611] alg: No test for adler32 (adler32-zlib) [ 2437.880486] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2438.080052] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2439.791354] Key type lgssc registered [ 2441.155708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.922491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2448.947971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.582327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.473243] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2460.129739] LustreError: 76944:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.151374] LustreError: 76944:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2463.947373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.480642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.200728] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2476.934815] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2481.903717] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2489.221864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.670908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.787143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2496.586552] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2502.502684] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2519.527416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2519.530567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.545200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.601805] Lustre: Skipped 1 previous similar message [ 2523.240302] Lustre: server umount lustre-OST0000 complete [ 2529.767624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.773421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.794362] Lustre: Skipped 1 previous similar message [ 2529.810800] Lustre: Skipped 2 previous similar messages [ 2533.501249] Lustre: server umount lustre-MDT0000 complete [ 2534.880132] LustreError: 77656:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.904305] LustreError: 77656:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2537.784208] LustreError: 76923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924637 with bad export cookie 2723793978635000113 [ 2537.793685] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.808961] LustreError: 76923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2538.149932] Lustre: server umount lustre-MDT0001 complete [ 2550.342188] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2553.700748] Key type lgssc unregistered [ 2554.030103] LNet: 80141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.035921] LNetError: 80141:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.109453] LNet: Removed LNI 192.168.203.130@tcp [ 2555.992451] Key type .llcrypt unregistered [ 2555.999897] Key type ._llcrypt unregistered [ 2580.425625] Key type ._llcrypt registered [ 2580.428913] Key type .llcrypt registered [ 2580.516773] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2581.430534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.581892] alg: No test for adler32 (adler32-zlib) [ 2582.599359] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2582.821887] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2584.575432] Key type lgssc registered [ 2585.594358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2593.016721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2593.039936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.609751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.166769] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2605.025862] LustreError: 81539:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.040230] LustreError: 81539:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2607.709486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.984706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2611.935795] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2617.532078] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2622.014037] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2629.805033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2630.122565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.426976] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2640.375821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2642.765228] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2650.595061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2650.596928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.607979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.618674] Lustre: Skipped 1 previous similar message [ 2650.629258] Lustre: Skipped 1 previous similar message [ 2655.718097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.729433] Lustre: Skipped 1 previous similar message [ 2656.146680] Lustre: server umount lustre-OST0000 complete [ 2660.833307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.834778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.851452] Lustre: Skipped 1 previous similar message [ 2660.864876] Lustre: Skipped 1 previous similar message [ 2665.870313] Lustre: server umount lustre-MDT0000 complete [ 2665.952839] LustreError: 82558:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.973301] LustreError: 82558:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2669.676050] LustreError: 81521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924769 with bad export cookie 10483379256332992752 [ 2669.684230] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.689826] LustreError: 81521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2670.052099] Lustre: server umount lustre-MDT0001 complete [ 2680.349805] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2682.838705] Key type lgssc unregistered [ 2683.107679] LNet: 84713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2683.115093] LNetError: 84713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2683.137094] LNet: Removed LNI 192.168.203.130@tcp [ 2684.186971] Key type .llcrypt unregistered [ 2684.195407] Key type ._llcrypt unregistered [ 2711.668760] Key type ._llcrypt registered [ 2711.674058] Key type .llcrypt registered [ 2711.827580] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2713.252054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.291632] alg: No test for adler32 (adler32-zlib) [ 2714.249319] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2714.408512] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2716.039166] Key type lgssc registered [ 2716.873732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2725.392453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2725.429127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.060463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.175230] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2737.636197] LustreError: 86110:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.674464] LustreError: 86110:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2738.892287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.197317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.144261] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2748.773807] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2753.134461] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2761.128554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.583052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.840888] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2772.857664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2774.188674] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2788.328533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2788.342280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.366575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2788.371813] Lustre: Skipped 1 previous similar message [ 2793.451429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.458192] Lustre: Skipped 1 previous similar message [ 2798.565796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.588125] Lustre: Skipped 1 previous similar message [ 2802.143146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2802.501594] Lustre: server umount lustre-OST0000 complete [ 2808.801384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.805135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.817445] Lustre: Skipped 2 previous similar messages [ 2808.843220] Lustre: Skipped 1 previous similar message [ 2813.238592] Lustre: server umount lustre-MDT0000 complete [ 2813.924252] LustreError: 86111:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.949966] LustreError: 86111:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2817.375780] LustreError: 86091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924917 with bad export cookie 14641699660479665400 [ 2817.377270] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2817.382631] LustreError: 86091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2817.637477] Lustre: server umount lustre-MDT0001 complete [ 2831.785374] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2834.410683] Key type lgssc unregistered [ 2834.703968] LNet: 89310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2834.715835] LNetError: 89310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2834.737830] LNet: Removed LNI 192.168.203.130@tcp [ 2835.681395] Key type .llcrypt unregistered [ 2835.685967] Key type ._llcrypt unregistered [ 2862.304480] Key type ._llcrypt registered [ 2862.307268] Key type .llcrypt registered [ 2862.398558] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2863.368038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2863.473109] alg: No test for adler32 (adler32-zlib) [ 2864.560526] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2864.819694] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2866.487158] Key type lgssc registered [ 2867.419916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2875.834904] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2875.863657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.487526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2896.608822] LustreError: 90679:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 2923.362548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.412490] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2935.847510] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2942.202763] Lustre: server umount lustre-OST0000 complete [ 2952.057819] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:50:50 (1778925050) [ 2959.106992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.623480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.959427] LustreError: 92206:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.972527] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2964.980883] LustreError: 92206:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2969.057675] LustreError: 92207:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.177514] LustreError: 92206:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.236120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.622059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2979.353134] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2985.793762] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.838337] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2996.953480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.406710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.290569] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3004.627450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3009.195332] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3025.381951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3025.383335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.392102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3025.392608] LustreError: Skipped 1 previous similar message [ 3025.431921] Lustre: Skipped 1 previous similar message [ 3028.279531] Lustre: server umount lustre-OST0000 complete [ 3035.617159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.627988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3035.631380] Lustre: Skipped 1 previous similar message [ 3035.635309] Lustre: Skipped 1 previous similar message [ 3038.848561] Lustre: server umount lustre-MDT0000 complete [ 3040.737311] LustreError: 93173:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.773662] LustreError: 93173:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3043.116244] LustreError: 93821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925143 with bad export cookie 12201589868111992674 [ 3043.117746] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.137696] LustreError: 93821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3043.562258] Lustre: server umount lustre-MDT0001 complete [ 3055.664260] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3058.839095] Key type lgssc unregistered [ 3059.187714] LNet: 95542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3059.206145] LNetError: 95542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3060.262988] LNet: Removed LNI 192.168.203.130@tcp [ 3061.110029] Key type .llcrypt unregistered [ 3061.113238] Key type ._llcrypt unregistered [ 3105.731972] Key type ._llcrypt registered [ 3105.737109] Key type .llcrypt registered [ 3105.859424] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3106.755554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3106.797573] alg: No test for adler32 (adler32-zlib) [ 3107.881462] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3108.108377] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3109.847159] Key type lgssc registered [ 3110.868288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3118.916996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3118.937966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.610436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.999656] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3130.851038] LustreError: 97710:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.872762] LustreError: 97710:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3135.970654] LustreError: 97711:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.053429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.348400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.098577] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3147.107856] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3151.162946] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3158.285413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3158.681083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.707647] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3165.883646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3172.313241] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3186.659824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.662614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.685640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.700080] Lustre: Skipped 1 previous similar message [ 3189.552630] Lustre: server umount lustre-OST0000 complete [ 3196.907271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.919445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.922125] Lustre: Skipped 2 previous similar messages [ 3196.938407] Lustre: Skipped 1 previous similar message [ 3199.879713] Lustre: server umount lustre-MDT0000 complete [ 3202.023432] LustreError: 97710:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.047112] LustreError: 97710:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3203.606373] LustreError: 97693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925303 with bad export cookie 1605099541973381801 [ 3203.610441] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.613235] LustreError: 97693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.843880] Lustre: server umount lustre-MDT0001 complete [ 3214.778598] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3217.511906] Key type lgssc unregistered [ 3217.761679] LNet: 100889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3217.773852] LNetError: 100889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3217.791237] LNet: Removed LNI 192.168.203.130@tcp [ 3218.445156] Key type .llcrypt unregistered [ 3218.447263] Key type ._llcrypt unregistered [ 3237.196211] Key type ._llcrypt registered [ 3237.197976] Key type .llcrypt registered [ 3237.308473] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3238.400617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3238.669157] alg: No test for adler32 (adler32-zlib) [ 3239.579847] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3239.753021] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3241.448399] Key type lgssc registered [ 3242.659642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3251.946284] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3251.962503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.551349] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3272.671474] LustreError: 102117:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 3299.359427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.050720] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3312.130893] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3320.092985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.352925] LustreError: MGC192.168.203.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3320.370539] LustreError: 103028:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3320.389259] LustreError: 103028:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3320.404859] LustreError: 103028:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3320.541363] Lustre: server umount lustre-MDT0000 complete [ 3320.554669] LustreError: 103028:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3341.792282] Lustre: 101455:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778925425/real 1778925425] req@0000000053a134bf x1865338416270592/t0(0) o400->MGC192.168.203.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778925441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3341.825379] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.851122] LustreError: 102129:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3342.034180] Lustre: server umount lustre-OST0000 complete [ 3352.761619] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3362.678259] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3400.564665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.047296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3401.111221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3401.310953] Lustre: lustre-MDT0000: new disk, initializing [ 3401.485911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.534703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3406.959567] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3417.815394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.919822] Lustre: 105787:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3417.979400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3417.984335] Lustre: Skipped 1 previous similar message [ 3418.051619] Lustre: lustre-MDT0001: new disk, initializing [ 3418.141620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3418.219853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3418.244794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3424.127640] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3433.623854] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3438.247759] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3446.318641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3446.615726] Lustre: lustre-OST0000: new disk, initializing [ 3446.619410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3446.626657] Lustre: 107528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3446.726485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.491461] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3453.044987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3453.063925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3453.263235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3460.140683] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3464.312706] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3464.571179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3469.089087] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3469.300673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3473.900365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.903354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.908699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3473.921698] Lustre: Skipped 1 previous similar message [ 3473.943848] LustreError: Skipped 1 previous similar message [ 3477.287197] Lustre: server umount lustre-OST0000 complete [ 3484.130891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.151815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.157474] Lustre: Skipped 2 previous similar messages [ 3489.249960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.262147] Lustre: Skipped 1 previous similar message [ 3493.344939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.391199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3495.711323] Lustre: server umount lustre-MDT0000 complete [ 3500.044474] LustreError: 105778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925600 with bad export cookie 1990861512916433219 [ 3500.057397] LustreError: 105778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3500.060474] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.604169] Lustre: server umount lustre-MDT0001 complete [ 3510.201142] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:00:08 (1778925608) [ 3514.363038] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3530.531274] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3541.305553] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3580.247481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.446931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3580.471236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3580.538644] Lustre: lustre-MDT0000: new disk, initializing [ 3580.613642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.637496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3585.307190] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3595.809377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.938597] Lustre: 112262:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3595.986188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3595.989474] Lustre: Skipped 1 previous similar message [ 3596.053454] Lustre: lustre-MDT0001: new disk, initializing [ 3596.161396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3596.207048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3596.218645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3600.567980] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3608.061596] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3611.305459] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3617.766591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.035481] Lustre: lustre-OST0000: new disk, initializing [ 3618.041234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3618.047905] Lustre: 114002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3618.146758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.874134] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3624.538698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3624.548718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3624.661265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3631.443465] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3636.105702] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3636.338795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3641.279935] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3641.452210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3645.411269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3645.415595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.445283] Lustre: Skipped 1 previous similar message [ 3645.460271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3645.471282] Lustre: Skipped 1 previous similar message [ 3649.264210] Lustre: server umount lustre-OST0000 complete [ 3655.650502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.653236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.674131] Lustre: Skipped 2 previous similar messages [ 3655.690624] Lustre: Skipped 2 previous similar messages [ 3659.823668] Lustre: server umount lustre-MDT0000 complete [ 3660.770353] LustreError: 112269:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.797897] LustreError: 112269:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 3664.122209] LustreError: 112254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925764 with bad export cookie 1990861512916434101 [ 3664.133786] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3664.535976] Lustre: server umount lustre-MDT0001 complete [ 3672.394907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.868709] LustreError: 116071:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.961827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.760384] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3678.183631] LustreError: 116072:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.277455] LustreError: 116071:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.675367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.121600] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3695.697106] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3700.049916] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3707.524769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3713.938768] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3719.963419] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3726.637267] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3731.426066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3731.432979] LustreError: Skipped 1 previous similar message [ 3731.438371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.458305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3732.969783] Lustre: server umount lustre-OST0000 complete [ 3737.057651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3751.391650] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3751.604119] Lustre: server umount lustre-MDT0000 complete [ 3752.929269] LustreError: 116072:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.942628] LustreError: 116072:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3755.826828] LustreError: 116053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925855 with bad export cookie 1990861512916434983 [ 3755.838420] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.841825] LustreError: 116053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3756.510782] Lustre: server umount lustre-MDT0001 complete [ 3769.396985] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3772.285944] Key type lgssc unregistered [ 3772.596992] LNet: 119346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3772.604763] LNetError: 119346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3773.670698] LNet: Removed LNI 192.168.203.130@tcp [ 3774.645353] Key type .llcrypt unregistered [ 3774.646898] Key type ._llcrypt unregistered [ 3794.944964] Key type ._llcrypt registered [ 3794.949363] Key type .llcrypt registered [ 3795.051427] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3810.992730] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3812.290761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3812.304221] alg: No test for adler32 (adler32-zlib) [ 3813.520719] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3813.817739] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3815.568423] Key type lgssc registered [ 3816.873896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3853.374372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3853.424907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.695142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3854.740836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3854.836401] Lustre: lustre-MDT0000: new disk, initializing [ 3854.987772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.016926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3859.744747] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3870.449873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3870.580487] Lustre: 122467:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3870.630557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3870.633995] Lustre: Skipped 1 previous similar message [ 3870.709441] Lustre: lustre-MDT0001: new disk, initializing [ 3870.830570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.881342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3870.898564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3875.068971] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3882.884426] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3886.711929] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3894.530969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.899257] Lustre: lustre-OST0000: new disk, initializing [ 3894.903868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3894.914838] Lustre: 124207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3895.012060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.481345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3900.499400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3900.652921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3901.218593] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3909.338621] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3914.729946] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3915.047185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3920.163808] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3920.386311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3926.500714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.501185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.509767] LustreError: Skipped 1 previous similar message [ 3926.521704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.532715] Lustre: Skipped 1 previous similar message [ 3929.088453] Lustre: server umount lustre-OST0000 complete [ 3936.741727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.760251] Lustre: Skipped 1 previous similar message [ 3936.766425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.484509] Lustre: server umount lustre-MDT0000 complete [ 3946.229263] LustreError: 122459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926046 with bad export cookie 17714661060126894909 [ 3946.229669] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.238662] LustreError: 122459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3946.827795] Lustre: server umount lustre-MDT0001 complete [ 3956.068779] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:07:34 (1778926054) [ 3962.825843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.331661] LustreError: 126475:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.347619] LustreError: 126475:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3963.393916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.948109] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3968.519802] LustreError: 126476:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.602912] LustreError: 126475:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3978.103265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.613939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3983.361413] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3989.058329] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3992.674854] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3994.745215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.760442] Lustre: Skipped 1 previous similar message [ 3994.768587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.771569] Lustre: Skipped 1 previous similar message [ 3999.200525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3999.209332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.222648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.125684] Lustre: server umount lustre-MDT0000 complete [ 4003.297205] LustreError: 126475:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.314362] LustreError: 126475:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4004.156652] LustreError: 126457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926104 with bad export cookie 17714661060126895791 [ 4004.160633] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.167527] LustreError: 126457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.563967] Lustre: server umount lustre-MDT0001 complete [ 4012.903774] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:08:31 (1778926111) [ 4019.820722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4035.423736] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4040.543893] LustreError: 128664:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 4066.271523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.307674] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4079.681599] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4082.263483] Lustre: server umount lustre-OST0000 complete [ 4091.332869] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:09:49 (1778926189) [ 4097.745332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.178300] LustreError: 129996:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.257652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.716281] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4108.772860] LustreError: 129997:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.798561] LustreError: 129997:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4111.184062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.440056] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4122.608184] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4126.953983] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4134.281953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.534430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.537731] Lustre: Skipped 1 previous similar message [ 4141.399831] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4148.820720] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4159.457728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.460529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4159.477644] Lustre: Skipped 1 previous similar message [ 4159.496968] Lustre: Skipped 4 previous similar messages [ 4164.354383] Lustre: server umount lustre-MDT0000 complete [ 4164.578674] LustreError: 129996:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.591201] LustreError: 129996:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4167.976942] LustreError: 129978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926267 with bad export cookie 17714661060126896449 [ 4167.982414] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4168.003817] LustreError: 129978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4168.230305] Lustre: server umount lustre-MDT0001 complete [ 4182.746193] Lustre: server umount lustre-OST0000 complete [ 4193.363666] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 06:11:31 (1778926291) [ 4201.268646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.791114] LustreError: 133312:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.865181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.179694] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4215.457963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.240396] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4226.280149] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4231.196178] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4238.617061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4245.712331] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4246.265946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4252.045718] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4257.088206] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4257.367455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4262.554294] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4262.794570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4266.977556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.988906] Lustre: Skipped 1 previous similar message [ 4266.989118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.993688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.263140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4279.593054] Lustre: server umount lustre-OST0000 complete [ 4287.456093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4287.460708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.462616] LustreError: Skipped 1 previous similar message [ 4287.463551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.463555] Lustre: Skipped 5 previous similar messages [ 4287.507319] Lustre: Skipped 2 previous similar messages [ 4290.287481] Lustre: server umount lustre-MDT0000 complete [ 4292.583864] LustreError: 133312:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.597387] LustreError: 133312:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4294.340542] LustreError: 136264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926394 with bad export cookie 17714661060126897527 [ 4294.353227] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.354876] LustreError: 136264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4294.719730] Lustre: server umount lustre-MDT0001 complete [ 4304.147878] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:13:22 (1778926402) [ 4311.007973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4326.559679] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4331.679872] LustreError: 137109:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 4357.407261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.416915] Lustre: Skipped 2 previous similar messages [ 4363.920371] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4369.846828] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4376.457664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.829151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4380.794665] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4391.608417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.112921] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4402.890415] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4407.308987] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4412.268359] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4412.400441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4417.049094] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4417.234275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4422.624343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4422.649517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.705575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4422.708376] Lustre: Skipped 1 previous similar message [ 4433.375117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4433.532396] Lustre: server umount lustre-MDT0000 complete [ 4433.896540] LustreError: 139198:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.934936] LustreError: 139198:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4437.820701] LustreError: 137117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926537 with bad export cookie 17714661060126898549 [ 4437.828288] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.846102] LustreError: 137117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4438.325055] Lustre: server umount lustre-MDT0001 complete [ 4453.187053] Lustre: server umount lustre-OST0000 complete [ 4463.702133] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 06:16:01 (1778926561) [ 4471.235515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.879746] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4492.003475] LustreError: 140932:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 4523.843408] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4529.618616] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4536.400541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4536.941567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4541.058165] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4549.397215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.051438] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4559.371494] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.278505] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4570.083470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.329440] Lustre: lustre-OST0001: new disk, initializing [ 4570.333647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4570.339736] Lustre: 143467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4576.233239] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4576.340320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4576.361383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4576.506172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4583.933620] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4588.197124] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4588.354161] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4592.413897] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4592.625506] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4596.195347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4596.207748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.221932] Lustre: Skipped 2 previous similar messages [ 4596.228849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4596.235269] Lustre: Skipped 6 previous similar messages [ 4600.578157] Lustre: server umount lustre-OST0000 complete [ 4610.625282] Lustre: server umount lustre-OST0001 complete [ 4616.674506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4616.687092] LustreError: Skipped 2 previous similar messages [ 4621.489711] Lustre: server umount lustre-MDT0000 complete [ 4624.987330] LustreError: 140938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926724 with bad export cookie 17714661060126899501 [ 4625.007150] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.403320] Lustre: server umount lustre-MDT0001 complete [ 4647.062836] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 06:19:05 (1778926745) [ 4648.677029] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4650.409297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4652.480715] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 06:19:10 (1778926750) [ 4659.081463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.429940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4659.457948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4659.733595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.738978] Lustre: Skipped 6 previous similar messages [ 4664.026928] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4672.219108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.343735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4672.372754] Lustre: 146878:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.105180] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4681.196905] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4685.291793] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4691.514358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.664630] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4697.262395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4697.634166] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4702.922932] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4707.414144] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4707.596906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4712.153713] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4712.411806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4717.546864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4720.378844] Lustre: server umount lustre-OST0000 complete [ 4743.105568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4744.433178] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.453039] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4749.675634] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4755.178067] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4758.823351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4758.830698] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4760.413961] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4760.574260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4765.050510] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4765.266669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4769.089267] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff942c48485000.ost_server_uuid 50 [ 4770.404472] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff942c48485000.ost_server_uuid in FULL state after 0 sec [ 4770.723339] LustreError: 149034:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.30@tcp) returned error from blocking AST (req@000000004ec2e25d x1865339018403200 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000028155307/0xf5d71e60bd4f0687 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.30@tcp remote: 0x578302fbfb7f5e52 expref: 5 pid: 146886 timeout: 4870 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4770.764834] LustreError: lustre-MDT0000: A client on nid 192.168.203.30@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4770.786164] LustreError: 146874:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.30@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000028155307/0xf5d71e60bd4f0687 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.30@tcp remote: 0x578302fbfb7f5e52 expref: 6 pid: 146886 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4775.393496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.413699] Lustre: Skipped 7 previous similar messages [ 4775.426137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4775.441885] Lustre: Skipped 10 previous similar messages [ 4781.440962] Lustre: server umount lustre-OST0000 complete [ 4785.121160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4785.128321] LustreError: Skipped 3 previous similar messages [ 4791.357223] Lustre: server umount lustre-MDT0000 complete [ 4794.849580] LustreError: 146885:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4794.869076] LustreError: 146885:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 10 previous similar messages [ 4794.954755] LustreError: 146869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926894 with bad export cookie 17714661060126900684 [ 4794.956400] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.966089] LustreError: 146869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4801.355413] Lustre: server umount lustre-MDT0001 complete [ 4810.350149] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4812.689958] Key type lgssc unregistered [ 4812.951596] LNet: 152353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4812.966281] LNetError: 152353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4812.982744] LNet: Removed LNI 192.168.203.130@tcp [ 4813.753322] Key type .llcrypt unregistered [ 4813.755112] Key type ._llcrypt unregistered [ 4833.171165] Key type ._llcrypt registered [ 4833.174760] Key type .llcrypt registered [ 4833.269148] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4834.141978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4834.234796] alg: No test for adler32 (adler32-zlib) [ 4835.299983] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 4835.502207] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4837.162781] Key type lgssc registered [ 4837.982582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4845.034653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4845.064401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.465198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.149826] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4856.808509] LustreError: 153728:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.831240] LustreError: 153728:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4857.788284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.064316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4861.846725] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4866.437586] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4869.442971] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4875.298310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4875.678875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.937453] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4885.890556] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4891.068774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4896.015672] Lustre: Failing over lustre-MDT0000 [ 4896.225985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.230754] Lustre: Skipped 1 previous similar message [ 4896.232696] LustreError: 153729:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.238467] LustreError: 153729:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4896.248077] Lustre: server umount lustre-MDT0000 complete [ 4901.348208] LustreError: 154440:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.365094] LustreError: 154440:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4906.474068] LustreError: 153728:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.503113] LustreError: 153728:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4907.860458] LustreError: 153711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.30@tcp arrived at 1778927007 with bad export cookie 8239141326115740715 [ 4911.589166] LustreError: 153729:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.603661] LustreError: 153729:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4913.684329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.862401] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4914.107513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.162631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4917.542485] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4919.251449] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4919.262704] Lustre: lustre-MDT0000: Denying connection for new client ca6d701c-b019-48e6-845c-813670da26f9 (at 192.168.203.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 4919.272443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4928.993634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4929.000943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4929.008949] Lustre: Skipped 1 previous similar message [ 4929.026207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.633544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.639425] Lustre: Skipped 3 previous similar messages [ 4939.747966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.756136] Lustre: Skipped 1 previous similar message [ 4941.279492] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4941.392342] Lustre: server umount lustre-OST0000 complete [ 4944.500537] LustreError: 157103:0:(ldlm_lib.c:2986:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4944.515084] Lustre: 156466:0:(ldlm_lib.c:2389:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4944.527057] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4944.548743] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4944.566787] LustreError: 156466:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001fcda2c2 x1865340089837952/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4944.592941] LustreError: 156466:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4944.604638] LustreError: 156466:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4944.614710] Lustre: lustre-MDT0000: Recovery over after 0:25, of 2 clients 0 recovered and 2 were evicted. [ 4944.866648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.877569] Lustre: Skipped 1 previous similar message [ 4944.886726] LustreError: 156805:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.971645] Lustre: server umount lustre-MDT0000 complete [ 4947.938356] LustreError: 153711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927047 with bad export cookie 8239141326115741044 [ 4947.939548] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.957064] LustreError: 153711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4947.960237] LustreError: 153037:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4950.315176] Lustre: server umount lustre-MDT0001 complete [ 4959.053716] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4961.218616] Key type lgssc unregistered [ 4961.483574] LNet: 157837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4961.492746] LNetError: 157837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4961.503655] LNet: Removed LNI 192.168.203.130@tcp [ 4962.417346] Key type .llcrypt unregistered [ 4962.423465] Key type ._llcrypt unregistered [ 4981.229577] Key type ._llcrypt registered [ 4981.231943] Key type .llcrypt registered [ 4981.337829] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4982.092579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4982.147067] alg: No test for adler32 (adler32-zlib) [ 4983.158993] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 4983.337542] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4984.983162] Key type lgssc registered [ 4986.003292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4992.452494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4992.468090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.832985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.942796] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5003.236734] LustreError: 159213:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.250020] LustreError: 159213:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5003.802933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.158095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5007.332789] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5011.810638] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5014.566458] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5019.479852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5019.762391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.282859] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5028.164668] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5038.188616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5043.681507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.681743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.691163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.717370] Lustre: Skipped 1 previous similar message [ 5045.261359] Lustre: server umount lustre-OST0000 complete [ 5048.806477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.808627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5048.822440] Lustre: Skipped 1 previous similar message [ 5048.828888] Lustre: Skipped 1 previous similar message [ 5053.921128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.925810] Lustre: Skipped 2 previous similar messages [ 5059.040814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5059.047475] Lustre: Skipped 1 previous similar message [ 5062.112171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5062.230854] Lustre: server umount lustre-MDT0000 complete [ 5063.137095] LustreError: 159208:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.150570] LustreError: 159208:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5064.700228] LustreError: 159194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927164 with bad export cookie 11125633774201697326 [ 5064.709115] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5064.981674] Lustre: server umount lustre-MDT0001 complete [ 5073.314853] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5075.456264] Key type lgssc unregistered [ 5075.731380] LNet: 162409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5075.741561] LNetError: 162409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5075.761211] LNet: Removed LNI 192.168.203.130@tcp [ 5076.410936] Key type .llcrypt unregistered [ 5076.414414] Key type ._llcrypt unregistered [ 5099.017765] Key type ._llcrypt registered [ 5099.019879] Key type .llcrypt registered [ 5099.090841] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5099.569646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5099.596052] alg: No test for adler32 (adler32-zlib) [ 5100.526108] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5100.669887] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5102.287158] Key type lgssc registered [ 5102.926382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5108.162481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5108.175623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.536575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.575582] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5118.626161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.923977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.066479] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5125.925417] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5128.671302] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5133.864924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.142301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.153855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5138.747704] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5142.804744] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.643378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5148.644109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.673584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.515061] Lustre: server umount lustre-OST0000 complete [ 5153.762230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.769938] Lustre: Skipped 1 previous similar message [ 5153.772812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5153.776872] Lustre: Skipped 2 previous similar messages [ 5158.056888] Lustre: server umount lustre-MDT0000 complete [ 5158.883640] LustreError: 166125:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.901129] LustreError: 166125:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5160.514560] LustreError: 164052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927260 with bad export cookie 3545067122507919286 [ 5160.516669] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.521747] LustreError: 164052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5160.750946] Lustre: server umount lustre-MDT0001 complete [ 5167.657438] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5169.448530] Key type lgssc unregistered [ 5169.627419] LNet: 167247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5169.637443] LNetError: 167247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5169.648512] LNet: Removed LNI 192.168.203.130@tcp [ 5170.125600] Key type .llcrypt unregistered [ 5170.127882] Key type ._llcrypt unregistered [ 5186.251758] Key type ._llcrypt registered [ 5186.254485] Key type .llcrypt registered [ 5186.323523] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5186.935302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.987631] alg: No test for adler32 (adler32-zlib) [ 5187.936031] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5188.079228] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5189.711408] Key type lgssc registered [ 5190.454183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5198.015180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5198.032609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.280523] Lustre: *** cfs_fail_loc=135, val=0*** [ 5199.332391] LustreError: 168732:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5199.336554] LustreError: 168732:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.130@tcp: cfg command failed: rc = -2 [ 5199.341618] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5199.341618] [ 5199.347910] LustreError: MGC192.168.203.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5199.360240] LustreError: 168693:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5199.366138] LustreError: 168693:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5199.372350] LustreError: 168693:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5199.377105] LustreError: 168693:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5199.454560] Lustre: server umount lustre-MDT0000 complete [ 5199.461257] LustreError: 168693:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5204.690313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.036552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.676182] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5213.410636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.658628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.313544] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5220.166611] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5223.002343] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5229.025981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5229.027610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.032822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.046541] Lustre: Skipped 1 previous similar message [ 5229.052730] Lustre: Skipped 1 previous similar message [ 5230.602889] Lustre: server umount lustre-MDT0000 complete [ 5233.046673] LustreError: 169083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927333 with bad export cookie 107549806322225690 [ 5233.051039] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.055079] LustreError: 169083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5233.253471] Lustre: server umount lustre-MDT0001 complete [ 5240.354220] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5242.146790] Key type lgssc unregistered [ 5242.322516] LNet: 171179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5242.326926] LNetError: 171179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5242.346206] LNet: Removed LNI 192.168.203.130@tcp [ 5242.825169] Key type .llcrypt unregistered [ 5242.828816] Key type ._llcrypt unregistered [ 5256.601313] Key type ._llcrypt registered [ 5256.604799] Key type .llcrypt registered [ 5256.675897] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5256.826601] Key type .llcrypt unregistered [ 5256.831277] Key type ._llcrypt unregistered [ 5268.581241] Key type ._llcrypt registered [ 5268.582879] Key type .llcrypt registered [ 5268.650548] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5269.270337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5269.282873] alg: No test for adler32 (adler32-zlib) [ 5270.222092] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5270.363645] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5271.999245] Key type lgssc registered [ 5272.822471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5278.105291] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5278.121937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5293.663286] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5298.784795] LustreError: 173270:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 5325.407748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.241542] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5333.342809] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5338.334556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.693931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.738057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5341.666541] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5346.274192] LustreError: 174196:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.284641] LustreError: 174196:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 5347.373606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.620354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5350.201270] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5353.585648] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5356.094534] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5358.151491] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5373.409653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.427362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5373.428655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5376.816312] Lustre: server umount lustre-OST0000 complete [ 5383.652150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5383.658722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5383.665061] Lustre: Skipped 1 previous similar message [ 5383.666411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.678251] Lustre: Skipped 2 previous similar messages [ 5385.431994] Lustre: server umount lustre-MDT0000 complete [ 5387.878475] LustreError: 173277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927487 with bad export cookie 17786243604274761921 [ 5387.880109] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.890166] LustreError: 173277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5388.045439] Lustre: server umount lustre-MDT0001 complete [ 5394.058603] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5395.764328] Key type lgssc unregistered [ 5395.964869] LNet: 177021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5395.973455] LNetError: 177021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5395.992266] LNet: Removed LNI 192.168.203.130@tcp [ 5396.485186] Key type .llcrypt unregistered [ 5396.487236] Key type ._llcrypt unregistered [ 5409.572768] Key type ._llcrypt registered [ 5409.574691] Key type .llcrypt registered [ 5409.641934] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5410.243149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5410.257031] alg: No test for adler32 (adler32-zlib) [ 5411.209586] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5411.367626] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5413.007143] Key type lgssc registered [ 5413.678796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5418.654241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5418.666196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.919341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.328184] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5427.275080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.514501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5429.691662] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5432.898614] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5435.067857] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5439.244607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5439.478591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.999495] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5444.587860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5446.093503] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5450.344306] Lustre: Failing over lustre-MDT0000 [ 5450.522882] Lustre: server umount lustre-MDT0000 complete [ 5452.450849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.453374] LustreError: 180462:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.464909] Lustre: Skipped 1 previous similar message [ 5452.478634] LustreError: 180462:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5453.783238] LustreError: 178390:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.791696] LustreError: 178390:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5457.888835] LustreError: 178394:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.900994] LustreError: 178394:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5463.010262] LustreError: 178395:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.020545] LustreError: 178395:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5465.326668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.398833] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5465.564121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5465.598769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5467.686603] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5469.130303] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5470.345098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5470.697538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5470.721542] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5470.768644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5471.637437] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5480.932361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.934027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5480.939054] Lustre: Skipped 2 previous similar messages [ 5486.049996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5486.054762] Lustre: Skipped 2 previous similar messages [ 5487.080580] Lustre: server umount lustre-OST0000 complete [ 5491.168592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.169369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5491.169657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.174324] Lustre: Skipped 1 previous similar message [ 5495.041705] Lustre: server umount lustre-MDT0000 complete [ 5496.288330] LustreError: 178391:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.299535] LustreError: 178391:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5496.994707] LustreError: 179113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927596 with bad export cookie 12277152374827374046 [ 5496.995683] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.004406] LustreError: 179113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5497.198981] Lustre: server umount lustre-MDT0001 complete [ 5502.305270] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5503.607587] Key type lgssc unregistered [ 5503.771823] LNet: 182756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5503.776455] LNetError: 182756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5503.791823] LNet: Removed LNI 192.168.203.130@tcp [ 5504.180504] Key type .llcrypt unregistered [ 5504.182277] Key type ._llcrypt unregistered [ 5516.680144] Key type ._llcrypt registered [ 5516.682013] Key type .llcrypt registered [ 5516.729823] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5517.247424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5517.265967] alg: No test for adler32 (adler32-zlib) [ 5518.225769] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5518.353623] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5519.959238] Key type lgssc registered [ 5520.621372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5525.015552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5525.031523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.331716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.456933] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5533.500460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.728140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5535.908617] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5539.024981] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5541.009804] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5544.580121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5544.746285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.589628] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5550.660969] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5551.917910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5557.916789] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5559.622919] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5566.711264] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5577.696887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5577.702262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.714248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5578.723169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.723467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5578.726773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.286821] Lustre: server umount lustre-OST0000 complete [ 5588.961467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.969102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5588.973219] Lustre: Skipped 2 previous similar messages [ 5591.640421] Lustre: server umount lustre-MDT0000 complete [ 5593.787731] LustreError: 184846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927693 with bad export cookie 5106020043792855540 [ 5593.793468] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5593.796798] LustreError: 184846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5593.996503] Lustre: server umount lustre-MDT0001 complete [ 5599.686823] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5601.051799] Key type lgssc unregistered [ 5601.210472] LNet: 187467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5601.215708] LNetError: 187467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5601.234953] LNet: Removed LNI 192.168.203.130@tcp [ 5601.660634] Key type .llcrypt unregistered [ 5601.662282] Key type ._llcrypt unregistered [ 5613.778865] Key type ._llcrypt registered [ 5613.780731] Key type .llcrypt registered [ 5613.830573] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5614.355963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5614.373729] alg: No test for adler32 (adler32-zlib) [ 5615.266945] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5615.398108] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5617.007380] Key type lgssc registered [ 5617.540787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5621.330626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5621.344303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.628908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.696234] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5629.053915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.270397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5631.454286] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5634.424657] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5636.336629] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5639.592178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.763902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.475142] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5645.080736] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5647.978509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5653.229859] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5655.076378] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5661.231125] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5670.938756] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5680.097412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.099280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5680.099440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5680.102771] Lustre: Skipped 1 previous similar message [ 5685.220872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.226966] Lustre: Skipped 2 previous similar messages [ 5685.855263] Lustre: server umount lustre-OST0000 complete [ 5690.336403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5690.337061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5690.342246] Lustre: Skipped 1 previous similar message [ 5693.801650] Lustre: server umount lustre-MDT0000 complete [ 5695.457447] LustreError: 188839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.467669] LustreError: 188839:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5695.504786] LustreError: 188819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927795 with bad export cookie 2722830434644409468 [ 5695.508892] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.510338] LustreError: 188819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5695.673872] Lustre: server umount lustre-MDT0001 complete [ 5700.466236] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5701.731457] Key type lgssc unregistered [ 5701.893351] LNet: 193592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5701.898184] LNetError: 193592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5701.911130] LNet: Removed LNI 192.168.203.130@tcp [ 5702.284314] Key type .llcrypt unregistered [ 5702.286018] Key type ._llcrypt unregistered [ 5713.793343] Key type ._llcrypt registered [ 5713.794805] Key type .llcrypt registered [ 5713.838158] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5714.318544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5714.330490] alg: No test for adler32 (adler32-zlib) [ 5715.228129] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5715.325132] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5716.919186] Key type lgssc registered [ 5717.388053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5720.809739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5720.822498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.034122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5723.721455] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5727.394885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.588799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.342778] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5731.822837] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5733.346436] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5736.146022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5736.297865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.700982] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5741.115415] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5747.557636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5752.800837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.801413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.804275] LustreError: Skipped 1 previous similar message [ 5752.808672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.813336] Lustre: Skipped 1 previous similar message [ 5753.498983] Lustre: server umount lustre-OST0000 complete [ 5757.099070] Lustre: server umount lustre-MDT0000 complete [ 5757.922476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.936570] Lustre: Skipped 1 previous similar message [ 5757.943018] LustreError: 197114:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.955061] LustreError: 197114:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5758.887734] LustreError: 194945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927858 with bad export cookie 2717391203111302735 [ 5758.889703] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5758.894155] LustreError: 194945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5759.032718] Lustre: server umount lustre-MDT0001 complete [ 5763.775489] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5765.108226] Key type lgssc unregistered [ 5765.265516] LNet: 198286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5765.270780] LNetError: 198286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5765.281176] LNet: Removed LNI 192.168.203.130@tcp [ 5765.647639] Key type .llcrypt unregistered [ 5765.649251] Key type ._llcrypt unregistered [ 5775.198414] Key type ._llcrypt registered [ 5775.200012] Key type .llcrypt registered [ 5775.250754] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5775.732034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.740517] alg: No test for adler32 (adler32-zlib) [ 5776.625742] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5776.723884] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5778.327161] Key type lgssc registered [ 5778.813258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5782.171204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5782.183605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.434291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.275841] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5789.756984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.036553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5792.385562] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5795.316581] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5797.066398] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5799.956805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.105264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.404723] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5804.896971] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5808.296522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5818.850507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5818.856430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.866506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.872950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.875431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.877353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.053219] Lustre: server umount lustre-OST0000 complete [ 5824.995088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.006960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.013580] Lustre: Skipped 2 previous similar messages [ 5830.114720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5830.122213] Lustre: Skipped 1 previous similar message [ 5835.232617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5835.236450] Lustre: Skipped 1 previous similar message [ 5838.303151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5838.389542] Lustre: server umount lustre-MDT0000 complete [ 5839.331318] LustreError: 199510:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.341436] LustreError: 199510:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5840.221034] LustreError: 201133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927940 with bad export cookie 14568879000589133253 [ 5840.224688] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5840.227854] LustreError: 201133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5840.392971] Lustre: server umount lustre-MDT0001 complete [ 5845.638220] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5847.072495] Key type lgssc unregistered [ 5847.234777] LNet: 202840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5847.239114] LNetError: 202840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.293556] LNet: Removed LNI 192.168.203.130@tcp [ 5848.685192] Key type .llcrypt unregistered [ 5848.686804] Key type ._llcrypt unregistered [ 5857.728211] Key type ._llcrypt registered [ 5857.730277] Key type .llcrypt registered [ 5857.779763] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5858.301560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5858.319414] alg: No test for adler32 (adler32-zlib) [ 5859.199358] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5859.307319] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5860.903160] Key type lgssc registered [ 5861.418985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5864.803472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.812111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.011384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.612515] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5871.390800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.598439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5873.486988] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5875.803623] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5877.420189] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5880.655970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5880.851660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.346957] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5885.732668] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5890.086388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5894.021971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5894.073541] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5894.190818] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5896.534124] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5898.817527] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5899.496651] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5910.139226] Lustre: Permanently deactivating lustre-OST0001 [ 5910.146068] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5917.182913] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5920.431773] Lustre: Permanently reactivating lustre-OST0001 [ 5920.439606] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5920.442304] Lustre: Skipped 2 previous similar messages [ 5923.334053] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.341225] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5923.344993] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5923.351805] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5923.364466] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5933.539110] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5933.539293] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.543187] LustreError: Skipped 1 previous similar message [ 5933.544356] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5933.549057] Lustre: Skipped 2 previous similar messages [ 5936.579657] Lustre: server umount lustre-OST0001 complete [ 5938.658929] LustreError: 205659:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.661883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.665143] LustreError: 205659:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5938.675873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5938.678700] Lustre: Skipped 1 previous similar message [ 5943.264160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5943.267160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.273400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5943.777110] LustreError: 205658:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.785342] LustreError: 205658:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5944.285847] Lustre: server umount lustre-OST0000 complete [ 5948.383974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5948.388262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.395125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.398241] Lustre: Skipped 1 previous similar message [ 5954.017972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5954.021540] Lustre: Skipped 2 previous similar messages [ 5960.159172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5960.239782] Lustre: server umount lustre-MDT0000 complete [ 5961.952287] LustreError: 204047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928061 with bad export cookie 6020021513873123803 [ 5961.955599] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5961.960212] LustreError: 204047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5962.089364] Lustre: server umount lustre-MDT0001 complete [ 5966.739064] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5968.007293] Key type lgssc unregistered [ 5968.162727] LNet: 208630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5968.167244] LNetError: 208630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5968.182079] LNet: Removed LNI 192.168.203.130@tcp [ 5968.547104] Key type .llcrypt unregistered [ 5968.548515] Key type ._llcrypt unregistered [ 5987.023738] Key type ._llcrypt registered [ 5987.025016] Key type .llcrypt registered [ 5987.072724] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5987.542462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5987.556094] alg: No test for adler32 (adler32-zlib) [ 5988.432668] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5988.547819] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5990.143218] Key type lgssc registered [ 5990.720087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5994.329689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5994.343037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.489583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5995.501339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5995.589135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5997.289684] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6001.104690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.163255] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6001.184467] Lustre: 210794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6001.346445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.060716] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6005.363320] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6007.103805] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6010.702826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6010.800881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6010.907654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.826732] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6016.337699] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6019.612973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6023.651916] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6027.243655] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6032.908205] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6039.644330] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6049.470491] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6066.676849] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6066.682104] Lustre: Skipped 2 previous similar messages [ 6100.290862] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6100.294750] Lustre: Skipped 3 previous similar messages [ 6165.578140] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6165.586925] Lustre: Skipped 7 previous similar messages [ 6183.393549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.394483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6183.396358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6183.400138] Lustre: Skipped 1 previous similar message [ 6183.411523] LustreError: Skipped 1 previous similar message [ 6188.514100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6188.518024] Lustre: Skipped 2 previous similar messages [ 6189.477894] Lustre: server umount lustre-OST0000 complete [ 6193.633666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.635500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6193.642497] Lustre: Skipped 1 previous similar message [ 6197.340942] Lustre: server umount lustre-MDT0000 complete [ 6198.960250] LustreError: 211517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928298 with bad export cookie 9856648434261397806 [ 6198.963073] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6198.966225] LustreError: 211517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6199.117190] Lustre: server umount lustre-MDT0001 complete [ 6203.138491] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6204.379717] Key type lgssc unregistered [ 6204.512463] LNet: 215218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6204.516290] LNetError: 215218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6204.529695] LNet: Removed LNI 192.168.203.130@tcp [ 6204.896372] Key type .llcrypt unregistered [ 6204.898429] Key type ._llcrypt unregistered [ 6215.053438] Key type ._llcrypt registered [ 6215.055712] Key type .llcrypt registered [ 6215.102223] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6215.455688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6215.501954] alg: No test for adler32 (adler32-zlib) [ 6216.414433] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6216.511990] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6218.103427] Key type lgssc registered [ 6218.573385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.749098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.757358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.976792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6224.653535] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6228.101269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.284853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.879447] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6231.998865] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6233.415754] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6236.100637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.290177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.537755] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6240.976573] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6247.590457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6259.168430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6259.171988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.178727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.266198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.266394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.266474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.269539] Lustre: Skipped 1 previous similar message [ 6265.124268] Lustre: server umount lustre-OST0000 complete [ 6268.384685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.386019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.390052] Lustre: Skipped 1 previous similar message [ 6273.504247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6273.506899] Lustre: Skipped 2 previous similar messages [ 6278.624030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6278.626350] Lustre: Skipped 1 previous similar message [ 6281.183294] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6281.253221] Lustre: server umount lustre-MDT0000 complete [ 6282.754411] LustreError: 218204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928382 with bad export cookie 10566619413978260352 [ 6282.755864] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.759780] LustreError: 218204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6282.879493] Lustre: server umount lustre-MDT0001 complete [ 6287.231939] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6288.482459] Key type lgssc unregistered [ 6288.636074] LNet: 219939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6288.640833] LNetError: 219939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6288.650747] LNet: Removed LNI 192.168.203.130@tcp [ 6289.023886] Key type .llcrypt unregistered [ 6289.025179] Key type ._llcrypt unregistered [ 6299.485942] Key type ._llcrypt registered [ 6299.487471] Key type .llcrypt registered [ 6299.537785] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6299.956982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6299.986944] alg: No test for adler32 (adler32-zlib) [ 6300.865957] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6300.974587] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6302.567168] Key type lgssc registered [ 6303.101851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6568.616222] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6569.875768] Key type lgssc unregistered [ 6570.034780] LNet: 221902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6570.038665] LNetError: 221902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6570.047658] LNet: Removed LNI 192.168.203.130@tcp [ 6570.414630] Key type .llcrypt unregistered [ 6570.416477] Key type ._llcrypt unregistered [ 6580.889242] Key type ._llcrypt registered [ 6580.891281] Key type .llcrypt registered [ 6580.945662] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6588.245729] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6588.645892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6588.684858] alg: No test for adler32 (adler32-zlib) [ 6589.565882] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6589.667678] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6591.263368] Key type lgssc registered [ 6591.791862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6604.873557] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6339 sec ========== 06:51:43 (1778928703) [ 6605.672388] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:51:44 (1778928704) === [ 6607.119593] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:51:46 (1778928706) === [ 6622.738544] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6624.020369] Key type lgssc unregistered [ 6624.163516] LNet: 227061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6624.166753] LNetError: 227061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6624.176526] LNet: Removed LNI 192.168.203.130@tcp [ 6624.520812] Key type .llcrypt unregistered [ 6624.522636] Key type ._llcrypt unregistered