[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472998489 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003048] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007987] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013080] *** VALIDATE selinux *** [ 0.020698] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025444] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028131] *** VALIDATE tmpfs *** [ 0.030334] *** VALIDATE proc *** [ 0.031251] *** VALIDATE cgroup *** [ 0.032025] *** VALIDATE cgroup2 *** [ 0.033278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.039974] debug: unmapping init [mem 0xffffffff89c59000-0xffffffff89c60fff] [ 0.042193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043641] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051356] rcu: Hierarchical SRCU implementation. [ 0.053592] smp: Bringing up secondary CPUs ... [ 0.054635] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.059014] smp: Brought up 1 node, 4 CPUs [ 0.061021] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137027] node 0 deferred pages initialised in 71ms [ 0.140090] devtmpfs: initialized [ 0.141250] x86/mm: Memory block size: 128MB [ 0.143873] gcov: version magic: 0x41383552 [ 0.148318] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150299] pinctrl core: initialized pinctrl subsystem [ 0.153412] [ 0.153983] ************************************************************* [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158016] ** ** [ 0.160012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162011] ** ** [ 0.164013] ** This means that this kernel is built to expose internal ** [ 0.166008] ** IOMMU data structures, which may compromise security on ** [ 0.168011] ** your system. ** [ 0.170012] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.175013] ** kernel, report this immediately to your vendor! ** [ 0.177010] ** ** [ 0.180015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182011] ************************************************************* [ 0.184723] NET: Registered protocol family 16 [ 0.186645] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197083] cpuidle: using governor menu [ 0.198732] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201385] PCI: Using configuration type 1 for base access [ 0.204135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213042] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214078] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216046] cryptd: max_cpu_qlen set to 1000 [ 0.219290] ACPI: Added _OSI(Module Device) [ 0.220017] ACPI: Added _OSI(Processor Device) [ 0.222014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223095] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230488] ACPI: Interpreter enabled [ 0.231000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232026] ACPI: Using IOAPIC for interrupt routing [ 0.234144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249113] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260407] acpiphp: Slot [2] registered [ 0.262091] acpiphp: Slot [5] registered [ 0.263107] acpiphp: Slot [6] registered [ 0.264152] acpiphp: Slot [7] registered [ 0.266084] acpiphp: Slot [8] registered [ 0.267093] acpiphp: Slot [9] registered [ 0.269125] acpiphp: Slot [10] registered [ 0.270095] acpiphp: Slot [3] registered [ 0.271156] acpiphp: Slot [4] registered [ 0.273082] acpiphp: Slot [11] registered [ 0.275074] acpiphp: Slot [12] registered [ 0.275827] acpiphp: Slot [13] registered [ 0.275880] acpiphp: Slot [14] registered [ 0.278104] acpiphp: Slot [15] registered [ 0.279118] acpiphp: Slot [16] registered [ 0.280090] acpiphp: Slot [17] registered [ 0.282121] acpiphp: Slot [18] registered [ 0.283113] acpiphp: Slot [19] registered [ 0.285098] acpiphp: Slot [20] registered [ 0.286081] acpiphp: Slot [21] registered [ 0.287088] acpiphp: Slot [22] registered [ 0.287954] acpiphp: Slot [23] registered [ 0.289053] acpiphp: Slot [24] registered [ 0.290062] acpiphp: Slot [25] registered [ 0.292114] acpiphp: Slot [26] registered [ 0.293099] acpiphp: Slot [27] registered [ 0.295141] acpiphp: Slot [28] registered [ 0.297122] acpiphp: Slot [29] registered [ 0.298109] acpiphp: Slot [30] registered [ 0.299082] acpiphp: Slot [31] registered [ 0.300053] PCI host bridge to bus 0000:00 [ 0.301016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.305020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.307024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.316075] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319280] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.333512] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336496] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347561] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357420] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.377029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.383014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389294] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430738] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.438020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.476861] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.483019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.490021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.514903] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.520017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.527018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.557029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565922] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.578018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.587019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.602014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.611207] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.627014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.651741] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.654367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.657461] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.659348] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.663241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.668097] iommu: Default domain type: Passthrough [ 0.669417] SCSI subsystem initialized [ 0.671148] ACPI: bus type USB registered [ 0.673097] usbcore: registered new interface driver usbfs [ 0.674068] usbcore: registered new interface driver hub [ 0.676102] usbcore: registered new device driver usb [ 0.678124] pps_core: LinuxPPS API ver. 1 registered [ 0.679007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.682058] PTP clock support registered [ 0.684040] EDAC MC: Ver: 3.0.0 [ 0.685458] PCI: Using ACPI for IRQ routing [ 0.688081] NetLabel: Initializing [ 0.690011] NetLabel: domain hash size = 128 [ 0.691012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.693091] NetLabel: unlabeled traffic allowed by default [ 0.696145] vgaarb: loaded [ 0.697167] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.699013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.706440] clocksource: Switched to clocksource kvm-clock [ 0.811114] VFS: Disk quotas dquot_6.6.0 [ 0.812172] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814164] *** VALIDATE ramfs *** [ 0.815608] *** VALIDATE hugetlbfs *** [ 0.817274] pnp: PnP ACPI init [ 0.819848] pnp: PnP ACPI: found 6 devices [ 0.836821] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.840249] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.842602] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.844974] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.847795] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.850285] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.852971] NET: Registered protocol family 2 [ 0.854953] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.859553] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.863333] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.868983] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.872627] TCP: Hash tables configured (established 65536 bind 65536) [ 0.875413] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878226] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881385] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884104] NET: Registered protocol family 1 [ 0.886629] RPC: Registered named UNIX socket transport module. [ 0.888395] RPC: Registered udp transport module. [ 0.889605] RPC: Registered tcp transport module. [ 0.890772] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.892507] NET: Registered protocol family 44 [ 0.893665] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.895175] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.896609] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.898237] PCI: CLS 0 bytes, default 64 [ 0.899380] Unpacking initramfs... [ 2.226138] debug: unmapping init [mem 0xffff89adfcc54000-0xffff89adfffbffff] [ 2.229157] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.230691] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.232527] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.721577] Initialise system trusted keyrings [ 2.722938] Key type blacklist registered [ 2.724401] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.732661] zbud: loaded [ 2.735743] *** VALIDATE nfs *** [ 2.737043] *** VALIDATE nfs4 *** [ 2.738439] pstore: using deflate compression [ 2.742438] Platform Keyring initialized [ 2.835160] NET: Registered protocol family 38 [ 2.836436] Key type asymmetric registered [ 2.837427] Asymmetric key parser 'x509' registered [ 2.838549] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.840601] io scheduler mq-deadline registered [ 2.841723] io scheduler kyber registered [ 2.842976] io scheduler bfq registered [ 2.844274] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.846355] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.848619] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.850396] ACPI: Power Button [PWRF] [ 2.854337] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.858976] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.866548] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.872372] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.885497] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.911590] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.939725] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.943457] Non-volatile memory driver v1.3 [ 2.945280] Linux agpgart interface v0.103 [ 2.969723] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.971493] vda: detected capacity change from 0 to 74694656 [ 2.985697] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.988944] vdb: detected capacity change from 0 to 1073741824 [ 3.005721] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.008756] vdc: detected capacity change from 0 to 2621440000 [ 3.024699] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.027440] vdd: detected capacity change from 0 to 2621440000 [ 3.041160] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.043260] vde: detected capacity change from 0 to 4294967296 [ 3.058589] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.060677] vdf: detected capacity change from 0 to 4294967296 [ 3.066250] libphy: Fixed MDIO Bus: probed [ 3.074907] usbcore: registered new interface driver usbserial_generic [ 3.076516] usbserial: USB Serial support registered for generic [ 3.077915] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.080723] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.081784] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.083591] mousedev: PS/2 mouse device common for all mice [ 3.086309] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.088309] rtc_cmos 00:05: RTC can wake from S4 [ 3.091349] rtc_cmos 00:05: registered as rtc0 [ 3.092092] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.093038] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.096382] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.097322] intel_pstate: CPU model not supported [ 3.102151] hid: raw HID events driver (C) Jiri Kosina [ 3.103546] usbcore: registered new interface driver usbhid [ 3.104960] usbhid: USB HID core driver [ 3.105884] drop_monitor: Initializing network drop monitor service [ 3.107438] Initializing XFRM netlink socket [ 3.108712] NET: Registered protocol family 10 [ 3.111072] Segment Routing with IPv6 [ 3.112119] NET: Registered protocol family 17 [ 3.113686] mpls_gso: MPLS GSO support [ 3.118347] RAS: Correctable Errors collector initialized. [ 3.119694] AVX version of gcm_enc/dec engaged. [ 3.121025] AES CTR mode by8 optimization enabled [ 3.178102] sched_clock: Marking stable (3178081690, 0)->(4055270852, -877189162) [ 3.180810] registered taskstats version 1 [ 3.182291] Loading compiled-in X.509 certificates [ 3.184001] zswap: loaded using pool lzo/zbud [ 3.200557] Key type big_key registered [ 3.212208] Key type encrypted registered [ 3.213194] ima: No TPM chip found, activating TPM-bypass! [ 3.214382] ima: Allocated hash algorithm: sha1 [ 3.215524] ima: No architecture policies found [ 3.216805] evm: Initialising EVM extended attributes: [ 3.218010] evm: security.selinux [ 3.218716] evm: security.ima [ 3.219452] evm: security.capability [ 3.220270] evm: HMAC attrs: 0x1 [ 3.222087] rtc_cmos 00:05: setting system clock to 2026-08-07 00:26:27 UTC (1786062387) [ 3.228160] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.231649] debug: unmapping init [mem 0xffffffff89982000-0xffffffff89c58fff] [ 3.242519] Write protecting the kernel read-only data: 28672k [ 3.244842] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.246635] debug: unmapping init [mem 0xffffffff88914000-0xffffffff889fffff] [ 3.273643] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.281660] systemd[1]: Detected virtualization kvm. [ 3.283569] systemd[1]: Detected architecture x86-64. [ 3.285096] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.308608] systemd[1]: No hostname configured. [ 3.309767] systemd[1]: Set hostname to . [ 3.311278] random: systemd: uninitialized urandom read (16 bytes read) [ 3.313530] systemd[1]: Initializing machine ID from random generator. [ 3.348543] random: ln: uninitialized urandom read (6 bytes read) [ 3.426419] random: systemd: uninitialized urandom read (16 bytes read) [ 3.428329] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.432743] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.435772] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.892992] device-mapper: uevent: version 1.0.3 [ 3.894526] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization.[ 4.444332] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.506065] virtio_net virtio0 ens2: renamed from eth0 [ 4.531950] scsi host0: ata_piix [ 4.537233] scsi host1: ata_piix [ 4.538770] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.541284] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.104043] dracut-initqueue[579]: RTNETLINK answers: File exists [ 8.725256] random: crng init done [ 8.726625] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.299854] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.309566] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.520383] SELinux: Disabled at runtime. [ 10.574921] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.580278] systemd[1]: Detected virtualization kvm. [ 10.581649] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.001485] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.003566] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.006541] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.009357] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.012702] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.020840] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.026641] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ 11.069799] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.401987] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.669189] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.699372] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.827279] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.841679] EDAC sbridge: Ver: 1.1.2 [ 13.297394] Key type dns_resolver registered [ 13.588618] NFS: Registering the id_resolver key type [ 13.589872] Key type id_resolver registered [ 13.590962] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg124-server login: [ 44.850361] libcfs: loading out-of-tree module taints kernel. [ 44.871656] Key type ._llcrypt registered [ 44.873116] Key type .llcrypt registered [ 44.922527] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 54.874572] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 55.631203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.662986] alg: No test for adler32 (adler32-zlib) [ 56.748632] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 57.189705] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 58.855300] Key type lgssc registered [ 61.237520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.784750] hrtimer: interrupt took 6740881 ns [ 84.520555] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.525683] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 146.320871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.349591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.497679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.527934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.599995] Lustre: lustre-MDT0000: new disk, initializing [ 147.731654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.751749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.828942] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.691548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.749712] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.804727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.809399] Lustre: Skipped 1 previous similar message [ 164.917546] Lustre: lustre-MDT0001: new disk, initializing [ 164.952051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.968229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.984501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.090779] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.788593] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.016419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.315357] Lustre: lustre-OST0000: new disk, initializing [ 185.318036] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.325678] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.418115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.646252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.657088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.758829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.887879] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.192529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.386231] Lustre: lustre-OST0001: new disk, initializing [ 212.389596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.394936] Lustre: 9495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.544890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.839307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.867297] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.997535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.147227] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.231828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.513315] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.189046] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.479361] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing yml_node [ 263.005280] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 265.888249] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 269.566366] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 271.477136] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 20:30:53 EDT 2026 [ 289.286681] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 290.503474] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.191906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.198849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.230122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.217141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.218347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.467340] Lustre: server umount lustre-MDT0000 complete [ 316.184514] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062700 with bad export cookie 7957878823476319356 [ 316.185760] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.198903] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 319.457033] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.458764] LustreError: 9963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.478738] Lustre: Skipped 2 previous similar messages [ 319.479556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.479561] Lustre: Skipped 3 previous similar messages [ 319.529617] LustreError: 9963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 321.504176] LustreError: 6495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.509476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.523280] LustreError: 6495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 321.527614] Lustre: Skipped 2 previous similar messages [ 322.630290] Lustre: server umount lustre-MDT0001 complete [ 326.414436] Lustre: server umount lustre-OST0000 complete [ 329.933803] Lustre: server umount lustre-OST0001 complete [ 336.845554] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 348.074304] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 383.543462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.821667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.854395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 383.987633] Lustre: lustre-MDT0000: new disk, initializing [ 384.156496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.216398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.957275] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 398.972181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.077700] Lustre: 15052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.131720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.135850] Lustre: Skipped 1 previous similar message [ 399.214192] Lustre: lustre-MDT0001: new disk, initializing [ 399.305585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.350985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.361377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 403.842291] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 412.864449] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.165053] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.526693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.853278] Lustre: lustre-OST0000: new disk, initializing [ 424.857179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.863188] Lustre: 17018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 424.992790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.947702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.960878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.098793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.082993] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 446.317883] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 451.904639] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 452.177745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 458.511734] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 458.772343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.816735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.818674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.826131] LustreError: Skipped 1 previous similar message [ 462.839335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.848174] Lustre: Skipped 2 previous similar messages [ 462.872720] Lustre: Skipped 1 previous similar message [ 467.281221] Lustre: server umount lustre-OST0000 complete [ 471.014225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.023081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.039637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.343674] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.689702] Lustre: server umount lustre-MDT0000 complete [ 488.417498] LustreError: 15060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.441763] LustreError: 15060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 489.332271] LustreError: 15044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062873 with bad export cookie 7957878823476320882 [ 489.337288] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.346587] LustreError: 15044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 489.922432] Lustre: server umount lustre-MDT0001 complete [ 493.953512] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:34:35 (1786062875) [ 501.570137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.014928] LustreError: 19288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.113618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.308896] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 507.362567] LustreError: 19289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.990436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.009825] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 533.065154] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.354479] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 548.884560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.445784] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 563.620985] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.212782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.217591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.259080] Lustre: Skipped 1 previous similar message [ 586.272675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.283031] Lustre: Skipped 6 previous similar messages [ 589.968055] Lustre: server umount lustre-OST0000 complete [ 597.484826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.492582] LustreError: Skipped 1 previous similar message [ 600.773741] Lustre: server umount lustre-MDT0000 complete [ 602.596841] LustreError: 19288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.625017] LustreError: 19288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 605.923963] LustreError: 19270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062990 with bad export cookie 7957878823476321771 [ 605.928959] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.938537] LustreError: 19270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 606.704875] Lustre: server umount lustre-MDT0001 complete [ 619.977383] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 623.200456] Key type lgssc unregistered [ 623.507939] LNet: 22845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.515480] LNetError: 22845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.526629] LNet: Removed LNI 192.168.201.124@tcp [ 624.636499] Key type .llcrypt unregistered [ 624.639898] Key type ._llcrypt unregistered [ 649.457507] Key type ._llcrypt registered [ 649.460254] Key type .llcrypt registered [ 649.562788] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 650.473933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.561556] alg: No test for adler32 (adler32-zlib) [ 651.690981] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 652.040117] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 653.735194] Key type lgssc registered [ 654.914948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 662.864502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.893904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.547845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.095969] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 673.774654] LustreError: 24256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.793147] LustreError: 24256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 678.880884] LustreError: 24257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.399463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.817600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 684.147941] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 691.526218] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 698.966505] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 708.603441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.351606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.439141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 717.597887] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 725.139925] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 745.449832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.459726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 745.472521] Lustre: Skipped 1 previous similar message [ 745.474526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 745.478510] Lustre: Skipped 1 previous similar message [ 747.765124] Lustre: server umount lustre-OST0000 complete [ 755.684237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.686560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.724097] Lustre: Skipped 1 previous similar message [ 755.742145] Lustre: Skipped 1 previous similar message [ 759.342899] Lustre: server umount lustre-MDT0000 complete [ 760.817378] LustreError: 25291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.833149] LustreError: 25291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 764.257222] LustreError: 24238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063148 with bad export cookie 6488125790308834857 [ 764.270182] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.271204] LustreError: 24238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 764.821669] Lustre: server umount lustre-MDT0001 complete [ 779.703098] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 783.309962] Key type lgssc unregistered [ 783.624795] LNet: 28239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.633796] LNetError: 28239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.691261] LNet: Removed LNI 192.168.201.124@tcp [ 785.862662] Key type .llcrypt unregistered [ 785.864740] Key type ._llcrypt unregistered [ 809.610058] Key type ._llcrypt registered [ 809.612210] Key type .llcrypt registered [ 809.678659] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 810.437659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.567113] alg: No test for adler32 (adler32-zlib) [ 811.672401] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 811.962307] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 813.668613] Key type lgssc registered [ 814.767441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.001092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.011538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.455987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.446215] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 833.001243] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.050588] LustreError: 29649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 836.868334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.306651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 843.626563] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 851.357968] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 858.245428] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.344099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.699578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.041255] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 886.723856] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 893.202845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 907.759324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 907.770851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.794956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.778777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.780218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.786624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.155151] Lustre: server umount lustre-OST0000 complete [ 919.008653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 919.009976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.018597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.026507] Lustre: Skipped 1 previous similar message [ 919.035155] Lustre: Skipped 2 previous similar messages [ 920.288424] Lustre: server umount lustre-MDT0000 complete [ 923.691758] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063307 with bad export cookie 6822167340280820696 [ 923.700593] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 923.703289] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 924.041629] Lustre: server umount lustre-MDT0001 complete [ 934.615489] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 937.089503] Key type lgssc unregistered [ 937.458726] LNet: 33553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.500643] LNetError: 33553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.549463] LNet: Removed LNI 192.168.201.124@tcp [ 938.536160] Key type .llcrypt unregistered [ 938.537596] Key type ._llcrypt unregistered [ 965.274798] Key type ._llcrypt registered [ 965.278041] Key type .llcrypt registered [ 965.447694] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 966.679731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.913933] alg: No test for adler32 (adler32-zlib) [ 968.095643] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 968.384517] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 970.119317] Key type lgssc registered [ 971.439740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 979.450982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 979.505617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.126548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.212642] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 991.728816] LustreError: 34966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.752986] LustreError: 34966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 994.405965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.806034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 999.551955] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1008.180866] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1014.380968] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1021.302544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1021.804121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.940053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1028.795810] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1036.855411] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1047.520650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.525264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.551570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.571519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.581579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.594439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.024910] Lustre: server umount lustre-OST0000 complete [ 1059.811389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.813385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.827385] Lustre: Skipped 1 previous similar message [ 1059.850201] Lustre: Skipped 2 previous similar messages [ 1063.438852] Lustre: server umount lustre-MDT0000 complete [ 1064.928148] LustreError: 34961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.985660] LustreError: 34961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1068.023885] LustreError: 37749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063452 with bad export cookie 7987610792176397452 [ 1068.033373] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1068.036917] LustreError: 37749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1068.511976] Lustre: server umount lustre-MDT0001 complete [ 1080.623265] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1083.428332] Key type lgssc unregistered [ 1083.759883] LNet: 38503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.769405] LNetError: 38503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.852493] LNet: Removed LNI 192.168.201.124@tcp [ 1085.805704] Key type .llcrypt unregistered [ 1085.808073] Key type ._llcrypt unregistered [ 1110.722386] Key type ._llcrypt registered [ 1110.726294] Key type .llcrypt registered [ 1110.817688] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1111.813993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.850479] alg: No test for adler32 (adler32-zlib) [ 1113.070805] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1113.495874] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1115.183195] Key type lgssc registered [ 1116.591673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.172477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.200875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.822807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.784425] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1138.151550] LustreError: 39921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.188181] LustreError: 39921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1142.759065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.401442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1148.770501] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1156.033420] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1161.593959] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.134098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.572481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.697888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1174.198967] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1183.019937] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.444236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.452843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.461530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.471639] Lustre: Skipped 1 previous similar message [ 1193.484522] Lustre: Skipped 1 previous similar message [ 1194.821856] Lustre: lustre-OST0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 1197.654354] Lustre: server umount lustre-OST0000 complete [ 1220.065608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1220.084720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.117653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.136132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.137159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.818767] Lustre: server umount lustre-MDT0000 complete [ 1228.256414] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.292986] LustreError: 39922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1230.027405] LustreError: 39901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063614 with bad export cookie 16722068266103012148 [ 1230.029617] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.048625] LustreError: 39901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1236.690680] Lustre: server umount lustre-MDT0001 complete [ 1247.545766] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1250.366268] Key type lgssc unregistered [ 1250.693573] LNet: 43630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.698279] LNetError: 43630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.723373] LNet: Removed LNI 192.168.201.124@tcp [ 1251.758389] Key type .llcrypt unregistered [ 1251.763374] Key type ._llcrypt unregistered [ 1275.525444] Key type ._llcrypt registered [ 1275.530447] Key type .llcrypt registered [ 1275.652470] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1276.715426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1276.967602] alg: No test for adler32 (adler32-zlib) [ 1278.029573] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1278.393440] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1280.087138] Key type lgssc registered [ 1281.378162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.356180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.383947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.885809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.128771] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1301.474157] LustreError: 45055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.490124] LustreError: 45055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1302.739668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.191113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.493496] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1315.183515] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1320.318699] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1326.982665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.305720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.010086] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1336.560895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1337.675531] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1345.341147] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 1347.050436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.051516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.075442] Lustre: Skipped 2 previous similar messages [ 1347.099940] Lustre: Skipped 2 previous similar messages [ 1350.426335] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 1351.180202] Lustre: server umount lustre-MDT0000 complete [ 1352.166809] LustreError: 45051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.176149] LustreError: 45051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1355.541578] LustreError: 46279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.708986] LustreError: 45034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063739 with bad export cookie 11212143599325925501 [ 1355.727983] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.731799] LustreError: 45034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1357.285211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.321961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1360.289627] LustreError: 46279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.321294] LustreError: 46279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1365.473995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1365.475567] LustreError: 45049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.478993] Lustre: Skipped 2 previous similar messages [ 1365.514558] LustreError: 45049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1370.079171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1370.313312] Lustre: server umount lustre-MDT0001 complete [ 1384.313539] Lustre: server umount lustre-OST0000 complete [ 1400.931774] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1403.285328] Key type lgssc unregistered [ 1403.593768] LNet: 48917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.599261] LNetError: 48917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.645971] LNet: Removed LNI 192.168.201.124@tcp [ 1405.663318] Key type .llcrypt unregistered [ 1405.665705] Key type ._llcrypt unregistered [ 1427.110273] Key type ._llcrypt registered [ 1427.111957] Key type .llcrypt registered [ 1427.191605] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1428.052871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1428.161583] alg: No test for adler32 (adler32-zlib) [ 1429.198514] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1429.529591] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1431.239169] Key type lgssc registered [ 1432.185037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1438.874331] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1438.888276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.433176] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1459.552802] LustreError: 50286:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 1486.239279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.343741] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1500.395794] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.124645] Lustre: server umount lustre-OST0000 complete [ 1689.385424] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1692.906548] Key type lgssc unregistered [ 1693.261824] LNet: 51966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.281397] LNetError: 51966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.305026] LNet: Removed LNI 192.168.201.124@tcp [ 1694.693858] Key type .llcrypt unregistered [ 1694.701269] Key type ._llcrypt unregistered [ 1721.508809] Key type ._llcrypt registered [ 1721.511120] Key type .llcrypt registered [ 1721.654049] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1723.083428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.127930] alg: No test for adler32 (adler32-zlib) [ 1724.197292] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1724.525668] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1726.320121] Key type lgssc registered [ 1727.641455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1737.015340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1737.025986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.586185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.818730] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1747.937308] LustreError: 53381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1747.965787] LustreError: 53381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1753.056795] LustreError: 53380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.864903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.422542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.331468] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1768.766277] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1774.863324] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1782.740911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.125323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.211378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1791.168923] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1799.682793] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1804.770331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.773558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1804.783402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.810982] Lustre: Skipped 1 previous similar message [ 1804.816568] LustreError: Skipped 1 previous similar message [ 1808.264813] Lustre: server umount lustre-OST0000 complete [ 1815.008422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.010972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.017404] Lustre: Skipped 1 previous similar message [ 1815.035593] Lustre: Skipped 2 previous similar messages [ 1819.224943] Lustre: server umount lustre-MDT0000 complete [ 1820.131434] LustreError: 53381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1820.156826] LustreError: 53381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1823.696193] LustreError: 53362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064207 with bad export cookie 10221370644986254577 [ 1823.702664] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.711046] LustreError: 53362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1824.130973] Lustre: server umount lustre-MDT0001 complete [ 1836.363937] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1839.554722] Key type lgssc unregistered [ 1839.841896] LNet: 56916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.852402] LNetError: 56916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.875985] LNet: Removed LNI 192.168.201.124@tcp [ 1842.125879] Key type .llcrypt unregistered [ 1842.127611] Key type ._llcrypt unregistered [ 1869.408226] Key type ._llcrypt registered [ 1869.409520] Key type .llcrypt registered [ 1869.533775] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1870.710093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.783818] alg: No test for adler32 (adler32-zlib) [ 1872.088513] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1872.414808] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1874.119193] Key type lgssc registered [ 1875.148586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1882.441689] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1882.460460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.017408] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1903.136038] LustreError: 58309:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 1929.887264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.939553] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1943.110657] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1949.400524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.863765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.931972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1954.437419] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1955.306168] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.324501] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1960.458539] LustreError: 58334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.189124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.465638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.557600] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1977.267788] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1985.086581] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1989.088702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1989.106152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.134311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.119174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.126809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.151235] Lustre: Skipped 1 previous similar message [ 1994.121435] Lustre: server umount lustre-OST0000 complete [ 2019.809284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2019.825440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.875734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2024.817768] Lustre: server umount lustre-MDT0000 complete [ 2025.958221] LustreError: 59351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.999886] LustreError: 59351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2028.916121] LustreError: 61302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064413 with bad export cookie 16751684219460693794 [ 2028.921459] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.930440] LustreError: 61302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2043.359442] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2043.843337] Lustre: server umount lustre-MDT0001 complete [ 2056.720833] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2059.937957] Key type lgssc unregistered [ 2060.349715] LNet: 62059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.359643] LNetError: 62059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.375949] LNet: Removed LNI 192.168.201.124@tcp [ 2061.357106] Key type .llcrypt unregistered [ 2061.362331] Key type ._llcrypt unregistered [ 2087.822709] Key type ._llcrypt registered [ 2087.825148] Key type .llcrypt registered [ 2087.923207] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2089.405103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.439873] alg: No test for adler32 (adler32-zlib) [ 2090.551710] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2090.938977] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2092.679220] Key type lgssc registered [ 2093.747493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.028912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.042735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.673876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.284753] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2112.995769] LustreError: 63488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.012434] LustreError: 63488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2118.121937] LustreError: 63489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.749934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.097394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2124.337430] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2133.402931] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2140.463253] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2146.593735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2146.872353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.896124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2153.196985] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2160.737665] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2179.553129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.559494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.562784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.578568] Lustre: Skipped 1 previous similar message [ 2179.596812] Lustre: Skipped 1 previous similar message [ 2184.678984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.690164] Lustre: Skipped 1 previous similar message [ 2189.795567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.809196] Lustre: Skipped 1 previous similar message [ 2193.375137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2193.690171] Lustre: server umount lustre-OST0000 complete [ 2200.036932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.039103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.048820] Lustre: Skipped 1 previous similar message [ 2204.950160] Lustre: server umount lustre-MDT0000 complete [ 2205.157565] LustreError: 65328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.184955] LustreError: 65328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2209.119040] LustreError: 63469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064593 with bad export cookie 15084887881145016588 [ 2209.132129] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.136817] LustreError: 63469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2209.516833] Lustre: server umount lustre-MDT0001 complete [ 2221.957575] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2225.598985] Key type lgssc unregistered [ 2226.026952] LNet: 67047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.034881] LNetError: 67047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.054272] LNet: Removed LNI 192.168.201.124@tcp [ 2227.227106] Key type .llcrypt unregistered [ 2227.233291] Key type ._llcrypt unregistered [ 2250.620382] Key type ._llcrypt registered [ 2250.625386] Key type .llcrypt registered [ 2250.731516] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2251.685515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.845291] alg: No test for adler32 (adler32-zlib) [ 2252.815623] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2253.074867] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2254.735640] Key type lgssc registered [ 2255.838494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2267.677064] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 21:04:09 (1786064649) [ 2273.973439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2273.995144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.627111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.858725] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2285.089467] LustreError: 68733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.118163] LustreError: 68733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2288.648450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.196224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.572928] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2301.797647] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2308.130085] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2314.706056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2315.166450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2320.329264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2321.170539] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2330.881291] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.163320] Lustre: Failing over lustre-MDT0000 [ 2341.545990] Lustre: server umount lustre-MDT0000 complete [ 2344.261930] LustreError: 68727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.447957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2345.452708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.474439] LustreError: 68732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.334111] LustreError: 68727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.371706] LustreError: 68727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2353.490884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.770747] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.086398] Lustre: *** cfs_fail_loc=135, val=0*** [ 2354.210020] LustreError: 71820:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2354.215131] LustreError: 71820:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -2 [ 2354.220662] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2354.220662] [ 2354.238070] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2354.259937] LustreError: 71808:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2354.278836] LustreError: 71808:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2354.292336] LustreError: 71808:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2354.299668] LustreError: 71808:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2354.337260] Lustre: server umount lustre-MDT0000 complete [ 2354.339105] LustreError: 71808:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2354.451704] LustreError: 70551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.478255] LustreError: 70551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2361.985448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.059751] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.230786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.275025] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2364.690931] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2367.033579] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2367.475508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2367.497195] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2367.545740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2376.672296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2376.677917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.694185] Lustre: Skipped 2 previous similar messages [ 2376.707497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.701985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.704773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.072536] Lustre: server umount lustre-OST0000 complete [ 2387.941103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.943393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2387.976500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.982443] Lustre: Skipped 2 previous similar messages [ 2393.056333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.060371] Lustre: Skipped 1 previous similar message [ 2398.176852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.185541] Lustre: Skipped 1 previous similar message [ 2399.199321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2399.637211] Lustre: server umount lustre-MDT0000 complete [ 2403.298386] LustreError: 68729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.318921] LustreError: 68729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2403.447473] LustreError: 68713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064787 with bad export cookie 6934268382549778690 [ 2403.461626] LustreError: 68713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2403.465812] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.819418] Lustre: server umount lustre-MDT0001 complete [ 2414.176554] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2416.998475] Key type lgssc unregistered [ 2417.285724] LNet: 73603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.302125] LNetError: 73603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.322982] LNet: Removed LNI 192.168.201.124@tcp [ 2418.187558] Key type .llcrypt unregistered [ 2418.190132] Key type ._llcrypt unregistered [ 2440.501398] Key type ._llcrypt registered [ 2440.504373] Key type .llcrypt registered [ 2440.621336] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2441.844533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.869933] alg: No test for adler32 (adler32-zlib) [ 2443.131933] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2443.514842] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2445.199298] Key type lgssc registered [ 2446.778636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2454.519821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2454.532703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.015101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.428761] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2466.272412] LustreError: 75008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.279845] LustreError: 75008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2468.782900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.200779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2473.601427] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2480.371567] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2485.705375] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2492.504804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.888547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.959971] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2501.119836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2506.622782] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2512.937131] Lustre: Failing over lustre-MDT0000 [ 2513.262809] Lustre: server umount lustre-MDT0000 complete [ 2514.235761] LustreError: 75005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.424630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2515.429973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.437078] LustreError: 75008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.467921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.326727] LustreError: 75003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.360400] LustreError: 75003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2521.105087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.195165] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.322085] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2521.458484] LustreError: 78093:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2521.470117] LustreError: 78093:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -9 [ 2521.480970] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2521.480970] [ 2521.487241] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2521.509133] LustreError: 78081:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2521.516605] LustreError: 78081:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2521.523547] LustreError: 78081:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2521.527206] LustreError: 78081:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2521.542804] Lustre: server umount lustre-MDT0000 complete [ 2521.544782] LustreError: 78081:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2524.437618] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.461573] LustreError: 75004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2527.313270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.428380] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.871615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.944327] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2529.561905] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2531.822625] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2533.363319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2533.390913] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2533.449134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2538.468331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.477846] Lustre: Skipped 2 previous similar messages [ 2538.482718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.475660] Lustre: server umount lustre-OST0000 complete [ 2548.704627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2548.706924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.720246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.731710] Lustre: Skipped 1 previous similar message [ 2548.746042] Lustre: Skipped 2 previous similar messages [ 2550.307925] Lustre: server umount lustre-MDT0000 complete [ 2553.834702] LustreError: 75009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.858765] LustreError: 75009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2553.880239] LustreError: 74989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064938 with bad export cookie 10161171359070348689 [ 2553.888528] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.889324] LustreError: 74989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2554.342320] Lustre: server umount lustre-MDT0001 complete [ 2564.035799] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2566.903805] Key type lgssc unregistered [ 2567.120252] LNet: 79874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2567.128057] LNetError: 79874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2567.144512] LNet: Removed LNI 192.168.201.124@tcp [ 2568.033894] Key type .llcrypt unregistered [ 2568.037993] Key type ._llcrypt unregistered [ 2583.333248] Key type ._llcrypt registered [ 2583.336844] Key type .llcrypt registered [ 2583.454552] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2583.930323] Key type .llcrypt unregistered [ 2583.932285] Key type ._llcrypt unregistered [ 2606.795370] Key type ._llcrypt registered [ 2606.797702] Key type .llcrypt registered [ 2606.894704] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2607.882380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2608.069846] alg: No test for adler32 (adler32-zlib) [ 2609.165920] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2609.370867] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2611.088713] Key type lgssc registered [ 2612.509604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.975635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.998543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.628701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.094947] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2632.166367] LustreError: 81783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.181099] LustreError: 81783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2634.994791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.251412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2639.310819] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2648.169019] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2654.245289] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.923150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.357353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.520097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2667.994843] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2675.181623] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2690.022086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.024903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.039750] Lustre: Skipped 1 previous similar message [ 2690.051545] Lustre: Skipped 1 previous similar message [ 2695.137985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.145940] Lustre: Skipped 1 previous similar message [ 2695.617728] Lustre: server umount lustre-OST0000 complete [ 2700.260932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.266490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.274717] Lustre: Skipped 1 previous similar message [ 2700.288628] Lustre: Skipped 1 previous similar message [ 2705.356755] Lustre: server umount lustre-MDT0000 complete [ 2705.379672] LustreError: 83896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.401652] LustreError: 83896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2708.485983] LustreError: 83627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065092 with bad export cookie 17521860201669924277 [ 2708.489593] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.492654] LustreError: 83627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2708.945299] Lustre: server umount lustre-MDT0001 complete [ 2721.108700] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2724.549957] Key type lgssc unregistered [ 2725.101380] LNet: 85345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2725.113751] LNetError: 85345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2725.132589] LNet: Removed LNI 192.168.201.124@tcp [ 2726.469243] Key type .llcrypt unregistered [ 2726.471523] Key type ._llcrypt unregistered [ 2756.048599] Key type ._llcrypt registered [ 2756.053172] Key type .llcrypt registered [ 2756.247307] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2758.063178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.138849] alg: No test for adler32 (adler32-zlib) [ 2759.502649] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2759.715681] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2761.447527] Key type lgssc registered [ 2763.036817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2771.696895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2771.710414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.237796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.126858] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2783.716761] LustreError: 86762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.746604] LustreError: 86762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2786.796229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.134805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2793.390480] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2801.811755] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2807.805872] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2815.266041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.634565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.739400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2822.570746] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2830.811878] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2837.473986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.479977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2837.491406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.498960] Lustre: Skipped 1 previous similar message [ 2839.019274] Lustre: server umount lustre-OST0000 complete [ 2843.621837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.632538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.645471] Lustre: Skipped 1 previous similar message [ 2843.656024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.714122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.730516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.174369] Lustre: server umount lustre-MDT0000 complete [ 2852.845322] LustreError: 88761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.869888] LustreError: 88761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2853.855547] LustreError: 86743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065238 with bad export cookie 3602767571605308780 [ 2853.861757] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.872738] LustreError: 86743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2854.193826] Lustre: server umount lustre-MDT0001 complete [ 2866.503548] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2870.325493] Key type lgssc unregistered [ 2870.646580] LNet: 90298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2870.653830] LNetError: 90298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2871.719389] LNet: Removed LNI 192.168.201.124@tcp [ 2872.704288] Key type .llcrypt unregistered [ 2872.706327] Key type ._llcrypt unregistered [ 2892.141522] Key type ._llcrypt registered [ 2892.145661] Key type .llcrypt registered [ 2892.271518] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2893.583958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2893.624835] alg: No test for adler32 (adler32-zlib) [ 2894.783590] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2894.956822] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2896.600851] Key type lgssc registered [ 2897.992817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2906.308133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2906.323599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.833159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.451411] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2918.375187] LustreError: 91692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.394061] LustreError: 91692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2921.944184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.290213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.205412] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2935.590980] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2941.976970] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2950.110450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2950.507516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.623330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2957.098840] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2964.876506] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2979.811660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2979.818500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.833721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.371028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.379146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.403186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.716289] Lustre: server umount lustre-OST0000 complete [ 2989.026066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2989.040688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.058670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.061523] Lustre: Skipped 1 previous similar message [ 2992.611719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.612715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.637690] Lustre: Skipped 1 previous similar message [ 2995.127585] Lustre: server umount lustre-MDT0000 complete [ 2997.729603] LustreError: 91692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.746031] LustreError: 91692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2999.301940] LustreError: 91673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065383 with bad export cookie 9501454324753987206 [ 2999.336816] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.030704] Lustre: server umount lustre-MDT0001 complete [ 3010.536245] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3013.155690] Key type lgssc unregistered [ 3013.382812] LNet: 95254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3013.396953] LNetError: 95254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3013.412866] LNet: Removed LNI 192.168.201.124@tcp [ 3014.098042] Key type .llcrypt unregistered [ 3014.099133] Key type ._llcrypt unregistered [ 3037.571744] Key type ._llcrypt registered [ 3037.575535] Key type .llcrypt registered [ 3037.707271] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3038.794192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3039.024852] alg: No test for adler32 (adler32-zlib) [ 3040.038967] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3040.267225] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3041.927193] Key type lgssc registered [ 3043.068983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.181804] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3051.206411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.783418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3071.903322] LustreError: 96640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 3098.593925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.493413] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3113.323787] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3119.706322] Lustre: server umount lustre-OST0000 complete [ 3127.486720] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:18:29 (1786065509) [ 3134.088759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.446939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.317129] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3139.563290] LustreError: 98279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.579809] LustreError: 98279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3144.684043] LustreError: 98280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.363831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.825048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3152.522505] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3160.903276] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3167.052397] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3173.378440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3173.986689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.162712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3182.334968] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3190.745890] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3204.072957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.082149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.082774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.110155] Lustre: Skipped 1 previous similar message [ 3204.145965] Lustre: Skipped 1 previous similar message [ 3207.647667] Lustre: server umount lustre-OST0000 complete [ 3214.316183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.329800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.334826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.341594] Lustre: Skipped 1 previous similar message [ 3218.429113] Lustre: server umount lustre-MDT0000 complete [ 3219.424389] LustreError: 98274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.440282] LustreError: 98274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3222.513474] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065606 with bad export cookie 17278290801915151118 [ 3222.519855] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.531549] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3222.882473] Lustre: server umount lustre-MDT0001 complete [ 3234.395054] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3237.319994] Key type lgssc unregistered [ 3237.592255] LNet: 101974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.598611] LNetError: 101974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3237.615176] LNet: Removed LNI 192.168.201.124@tcp [ 3238.772594] Key type .llcrypt unregistered [ 3238.775465] Key type ._llcrypt unregistered [ 3279.962515] Key type ._llcrypt registered [ 3279.975342] Key type .llcrypt registered [ 3280.167277] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3281.500761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3281.734876] alg: No test for adler32 (adler32-zlib) [ 3282.971429] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3283.261223] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3284.943858] Key type lgssc registered [ 3286.858832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3296.041499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3296.065754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.647100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.803377] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3307.043324] LustreError: 104159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3307.058812] LustreError: 104159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3311.343266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.670365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3316.655610] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3324.431557] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3330.722942] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3337.366723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.767153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.883612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3343.506847] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3350.691077] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3366.368292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3366.369720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.379307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.402489] Lustre: Skipped 1 previous similar message [ 3369.262514] Lustre: server umount lustre-OST0000 complete [ 3376.621727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.645180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3376.652117] Lustre: Skipped 1 previous similar message [ 3376.669418] Lustre: Skipped 2 previous similar messages [ 3379.986033] Lustre: server umount lustre-MDT0000 complete [ 3381.732124] LustreError: 104159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.752180] LustreError: 104159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3383.538778] LustreError: 104138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065767 with bad export cookie 5968583380573248102 [ 3383.541674] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3383.549477] LustreError: 104138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3383.939762] Lustre: server umount lustre-MDT0001 complete [ 3396.847470] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3399.451185] Key type lgssc unregistered [ 3399.826738] LNet: 107711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3399.832333] LNetError: 107711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3400.872149] LNet: Removed LNI 192.168.201.124@tcp [ 3401.850353] Key type .llcrypt unregistered [ 3401.853154] Key type ._llcrypt unregistered [ 3420.957418] Key type ._llcrypt registered [ 3420.959423] Key type .llcrypt registered [ 3421.034199] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3422.145579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3422.218108] alg: No test for adler32 (adler32-zlib) [ 3423.313704] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3423.738250] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3425.527239] Key type lgssc registered [ 3426.854644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3435.895613] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3435.911448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3451.488699] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3456.607413] LustreError: 108957:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 3483.359210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.071689] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3496.918337] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3503.811246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.122830] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3504.134653] LustreError: 109980:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3504.142091] LustreError: 109980:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3504.148983] LustreError: 109980:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3504.262640] Lustre: server umount lustre-MDT0000 complete [ 3504.265360] LustreError: 109980:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3525.599283] Lustre: 108293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786065893/real 1786065893] req@0000000084a918f9 x1872825739644160/t0(0) o400->MGC192.168.201.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786065909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3525.663332] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3525.709493] LustreError: 108970:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3546.249577] Lustre: server umount lustre-OST0000 complete [ 3558.114516] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3568.462262] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3612.727515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.883958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3612.910176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3612.967653] Lustre: lustre-MDT0000: new disk, initializing [ 3613.050865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.067168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3618.542789] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3630.192756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.308236] Lustre: 112775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3630.357931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3630.360854] Lustre: Skipped 1 previous similar message [ 3630.451669] Lustre: lustre-MDT0001: new disk, initializing [ 3630.586609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3630.621091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3630.636836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3634.764817] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3643.658687] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3651.107390] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3659.513566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.891370] Lustre: lustre-OST0000: new disk, initializing [ 3659.901820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3659.907686] Lustre: 114746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3660.058866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.021899] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3668.065895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3668.078358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3668.225451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3675.778185] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3680.164691] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3680.437822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3685.784756] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3686.060723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3688.931995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.934292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.940905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.954085] Lustre: Skipped 1 previous similar message [ 3694.056552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3694.063671] Lustre: Skipped 1 previous similar message [ 3694.530458] Lustre: server umount lustre-OST0000 complete [ 3699.168453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.192961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.197916] Lustre: Skipped 2 previous similar messages [ 3704.290001] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.296346] Lustre: Skipped 1 previous similar message [ 3705.059890] Lustre: server umount lustre-MDT0000 complete [ 3709.934874] LustreError: 112768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066094 with bad export cookie 20401728676968388 [ 3709.936187] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.943218] LustreError: 112768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3710.654667] Lustre: server umount lustre-MDT0001 complete [ 3720.587911] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:28:22 (1786066102) [ 3724.137354] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3739.701279] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3750.503218] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3784.730482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.953636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3784.988490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3785.066574] Lustre: lustre-MDT0000: new disk, initializing [ 3785.151221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.163655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3789.423565] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3800.874947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.098267] Lustre: 119644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3801.191931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3801.200615] Lustre: Skipped 1 previous similar message [ 3801.286129] Lustre: lustre-MDT0001: new disk, initializing [ 3801.371261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.414733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3801.435725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3806.051368] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3816.316238] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3821.869767] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3828.643485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.870371] Lustre: lustre-OST0000: new disk, initializing [ 3828.873886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3828.879803] Lustre: 121611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3828.978210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.986704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3835.006511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3835.193554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3836.167660] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3847.169567] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3853.145500] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3853.463183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3858.659544] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3859.036580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3860.965508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.983916] Lustre: Skipped 1 previous similar message [ 3860.989944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.992212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.994655] LustreError: Skipped 1 previous similar message [ 3861.009658] Lustre: Skipped 1 previous similar message [ 3871.201116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.212903] Lustre: Skipped 3 previous similar messages [ 3875.298686] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3875.466929] Lustre: server umount lustre-OST0000 complete [ 3881.441464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.450586] Lustre: Skipped 2 previous similar messages [ 3885.025417] Lustre: server umount lustre-MDT0000 complete [ 3886.560899] LustreError: 119654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.571203] LustreError: 119654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3889.982760] LustreError: 119637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066274 with bad export cookie 20401728676969270 [ 3889.995942] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3890.014830] LustreError: 119637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3891.241322] Lustre: server umount lustre-MDT0001 complete [ 3905.645074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.802955] LustreError: 123829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.994269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.170516] LustreError: 123830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.414774] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3917.285066] LustreError: 123829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.406852] LustreError: 123830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3925.578048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.073534] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3938.594651] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3944.677191] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3951.018383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3957.464642] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3964.370849] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3971.478750] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3978.219026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3978.226040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.250854] LustreError: Skipped 1 previous similar message [ 3978.251814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3978.251818] Lustre: Skipped 2 previous similar messages [ 3978.295684] Lustre: Skipped 1 previous similar message [ 3981.457505] Lustre: server umount lustre-OST0000 complete [ 3988.450061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.493540] Lustre: Skipped 1 previous similar message [ 3993.976632] Lustre: server umount lustre-MDT0000 complete [ 3998.556514] LustreError: 123810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066382 with bad export cookie 20401728676970152 [ 3998.562898] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3998.566861] LustreError: 123810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3999.519416] Lustre: server umount lustre-MDT0001 complete [ 4011.774640] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 4015.224299] Key type lgssc unregistered [ 4015.703741] LNet: 127463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4015.710395] LNetError: 127463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4015.746261] LNet: Removed LNI 192.168.201.124@tcp [ 4017.027621] Key type .llcrypt unregistered [ 4017.031349] Key type ._llcrypt unregistered [ 4036.301645] Key type ._llcrypt registered [ 4036.305885] Key type .llcrypt registered [ 4036.520638] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 4052.961198] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 4054.570616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4054.651291] alg: No test for adler32 (adler32-zlib) [ 4055.702794] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 4055.966303] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 4057.609102] Key type lgssc registered [ 4058.970890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4094.217489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4094.281838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.632416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4095.682984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4095.786808] Lustre: lustre-MDT0000: new disk, initializing [ 4095.973773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.009086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4101.364867] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4113.287117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.395382] Lustre: 130622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4113.434538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4113.439795] Lustre: Skipped 1 previous similar message [ 4113.489380] Lustre: lustre-MDT0001: new disk, initializing [ 4113.579495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.622384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4113.643675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4117.987882] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4127.611842] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4133.788976] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4140.535343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4140.772694] Lustre: lustre-OST0000: new disk, initializing [ 4140.776405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4140.782500] Lustre: 132589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4140.864619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.290577] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4148.773898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4148.795195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4148.920577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4156.090314] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4159.618675] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4159.787608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4165.022787] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4165.332564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4169.701060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.710164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.714344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.720541] Lustre: Skipped 1 previous similar message [ 4169.751879] Lustre: Skipped 1 previous similar message [ 4174.156785] Lustre: server umount lustre-OST0000 complete [ 4179.941496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.948131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.959649] Lustre: Skipped 1 previous similar message [ 4179.982718] Lustre: Skipped 1 previous similar message [ 4185.056421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.062521] Lustre: Skipped 1 previous similar message [ 4190.182333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4190.186207] Lustre: Skipped 1 previous similar message [ 4193.247248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4193.559863] Lustre: server umount lustre-MDT0000 complete [ 4197.809682] LustreError: 130612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066582 with bad export cookie 2705595640608353529 [ 4197.831603] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.326426] Lustre: server umount lustre-MDT0001 complete [ 4208.671853] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:36:30 (1786066590) [ 4216.916492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.743421] LustreError: 135001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.769134] LustreError: 135001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4217.902611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.814650] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4222.949688] LustreError: 135002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.065596] LustreError: 135001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.700567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.382361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4238.128340] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4247.232619] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4255.810533] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4258.285587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.298649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.315542] Lustre: Skipped 1 previous similar message [ 4258.330229] Lustre: Skipped 1 previous similar message [ 4264.195111] Lustre: server umount lustre-MDT0000 complete [ 4264.416476] LustreError: 135001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.431838] LustreError: 135001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4268.224537] LustreError: 136570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066652 with bad export cookie 2705595640608354411 [ 4268.232695] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.235255] LustreError: 136570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4268.555736] Lustre: server umount lustre-MDT0001 complete [ 4278.249081] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:37:40 (1786066660) [ 4284.379579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4300.191638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4305.311448] LustreError: 137417:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 4331.039914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.719227] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4344.614596] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4346.556271] Lustre: server umount lustre-OST0000 complete [ 4354.479567] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:38:56 (1786066736) [ 4360.539146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.926234] LustreError: 138872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.016844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.656883] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4371.430029] LustreError: 138872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.442227] LustreError: 138872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4373.486601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.474945] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4384.324509] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4389.431097] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4396.384263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.664253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.669396] Lustre: Skipped 1 previous similar message [ 4403.138888] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4410.102358] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4416.482497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.487315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4416.490625] Lustre: Skipped 1 previous similar message [ 4416.493908] Lustre: Skipped 2 previous similar messages [ 4422.121514] Lustre: server umount lustre-MDT0000 complete [ 4426.185583] LustreError: 138852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066810 with bad export cookie 2705595640608355076 [ 4426.193045] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4426.199247] LustreError: 138852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4426.625315] Lustre: server umount lustre-MDT0001 complete [ 4441.073157] Lustre: server umount lustre-OST0000 complete [ 4450.942349] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:40:33 (1786066833) [ 4457.804756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.337514] LustreError: 142524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.699509] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4471.226070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.837182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4471.849397] Lustre: Skipped 1 previous similar message [ 4477.007611] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4483.581226] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4489.071992] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4495.747356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4503.342066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4503.897129] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4514.035424] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4521.006604] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4521.300971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4524.939261] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4525.237167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4528.040907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.049624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4528.060166] Lustre: Skipped 2 previous similar messages [ 4528.066400] Lustre: Skipped 6 previous similar messages [ 4533.626070] Lustre: server umount lustre-OST0000 complete [ 4538.341504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4538.341683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.351534] LustreError: Skipped 1 previous similar message [ 4538.355660] Lustre: Skipped 1 previous similar message [ 4551.135161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4551.343152] Lustre: server umount lustre-MDT0000 complete [ 4553.696642] LustreError: 142524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4553.707149] LustreError: 142524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4554.984334] LustreError: 142506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066939 with bad export cookie 2705595640608356147 [ 4554.988390] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4555.001276] LustreError: 142506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4555.936547] Lustre: server umount lustre-MDT0001 complete [ 4566.481806] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:42:28 (1786066948) [ 4574.305961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4590.048057] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4595.167663] LustreError: 146686:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 4620.896885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.905465] Lustre: Skipped 1 previous similar message [ 4627.594780] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4635.923674] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4643.059291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.584262] LustreError: 146711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.734983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4648.761470] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4657.218166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.115429] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4669.807170] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4675.584738] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4681.335831] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4681.618569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4688.011318] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4688.155979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4693.478637] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.482177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4693.482345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4693.506152] Lustre: Skipped 2 previous similar messages [ 4693.546355] Lustre: Skipped 10 previous similar messages [ 4696.827434] Lustre: server umount lustre-MDT0000 complete [ 4702.031307] LustreError: 146694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067086 with bad export cookie 2705595640608357169 [ 4702.039651] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4702.388666] Lustre: server umount lustre-MDT0001 complete [ 4716.716197] Lustre: server umount lustre-OST0000 complete [ 4725.453382] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:45:07 (1786067107) [ 4733.675394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.538381] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4754.655460] LustreError: 150883:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 4786.718813] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4794.971610] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4803.091148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.684631] LustreError: 150908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.702510] LustreError: 150908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4803.837554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4808.310603] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4818.058552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.481367] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4833.437929] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4840.960484] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4848.377428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.546343] Lustre: lustre-OST0001: new disk, initializing [ 4848.548915] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4848.553066] Lustre: 153759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4854.306864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4854.316916] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4854.504360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4855.764481] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4865.253206] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4870.332971] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4870.609438] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4876.246354] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4876.485691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4879.877287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4879.883046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.889101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4884.808621] Lustre: server umount lustre-OST0000 complete [ 4890.086363] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4894.785674] Lustre: server umount lustre-OST0001 complete [ 4900.321700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4900.330621] LustreError: Skipped 1 previous similar message [ 4905.373896] Lustre: server umount lustre-MDT0000 complete [ 4909.509790] LustreError: 150891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067293 with bad export cookie 2705595640608358121 [ 4909.528076] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.529975] LustreError: 150891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4909.960686] Lustre: server umount lustre-MDT0001 complete [ 4938.687635] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:48:40 (1786067320) [ 4940.121184] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4941.591068] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4943.072645] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:48:45 (1786067325) [ 4951.116780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.747548] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4951.856547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4952.260635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.268523] Lustre: Skipped 6 previous similar messages [ 4957.463074] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4966.783993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.857154] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4966.880949] Lustre: 157319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4972.527466] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 4982.225811] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4989.476855] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4998.306980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.572876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5004.496237] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5005.861789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5012.887852] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5018.891535] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5019.134906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5023.845669] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5024.074378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5026.282210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5026.282363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.305683] LustreError: Skipped 1 previous similar message [ 5026.331495] Lustre: Skipped 6 previous similar messages [ 5026.346713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5026.356370] Lustre: Skipped 6 previous similar messages [ 5032.224239] Lustre: server umount lustre-OST0000 complete [ 5055.195039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.693916] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.718834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.729736] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5061.688266] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5069.698741] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5075.122187] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5075.506302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5080.382378] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5080.933618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5087.317958] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9788d0a0c000.ost_server_uuid 50 [ 5089.308865] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9788d0a0c000.ost_server_uuid in FULL state after 0 sec [ 5089.617835] LustreError: 158319:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.24@tcp) returned error from blocking AST (req@0000000068e6a703 x1872826403656320 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000066b50864/0x258c34e6784e8843 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.24@tcp remote: 0x3360926d782c3c5c expref: 5 pid: 158319 timeout: 5189 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5089.644593] LustreError: lustre-MDT0000: A client on nid 192.168.201.24@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5089.650751] LustreError: 157314:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.24@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000066b50864/0x258c34e6784e8843 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.24@tcp remote: 0x3360926d782c3c5c expref: 6 pid: 158319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5097.958079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.982149] LustreError: Skipped 1 previous similar message [ 5101.292478] Lustre: server umount lustre-OST0000 complete [ 5111.389364] Lustre: server umount lustre-MDT0000 complete [ 5113.313821] LustreError: 157326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.336517] LustreError: 157326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5115.479504] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067499 with bad export cookie 2705595640608359304 [ 5115.493076] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5115.496950] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5121.958485] Lustre: server umount lustre-MDT0001 complete [ 5132.757942] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5136.014071] Key type lgssc unregistered [ 5136.372715] LNet: 163438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5136.386903] LNetError: 163438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5136.408588] LNet: Removed LNI 192.168.201.124@tcp [ 5137.392711] Key type .llcrypt unregistered [ 5137.394273] Key type ._llcrypt unregistered [ 5163.374306] Key type ._llcrypt registered [ 5163.377348] Key type .llcrypt registered [ 5163.500183] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5165.300148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5165.354656] alg: No test for adler32 (adler32-zlib) [ 5166.356877] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5166.610482] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5168.319166] Key type lgssc registered [ 5169.320046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5178.295996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5178.315532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.153994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.954592] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5190.654752] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.686933] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5195.758268] LustreError: 164852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.036722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.546553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5203.138676] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5210.576724] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5215.417705] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5221.800659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5222.130831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.213956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5228.278469] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5235.343119] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5240.339460] Lustre: Failing over lustre-MDT0000 [ 5240.804546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.807721] LustreError: 164852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.812221] Lustre: Skipped 2 previous similar messages [ 5240.881886] Lustre: server umount lustre-MDT0000 complete [ 5245.920972] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.947310] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5251.043136] LustreError: 165891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.069884] LustreError: 165891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5253.596634] LustreError: 164831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.24@tcp arrived at 1786067637 with bad export cookie 3474341084105571783 [ 5260.777743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.968671] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.364500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5261.434870] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5265.941443] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5266.403392] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5266.440441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5268.886136] Lustre: lustre-MDT0000: Denying connection for new client 4221c49c-a201-4179-94cf-9538a88d7def (at 192.168.201.24@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5279.713132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5279.730133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.760535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5281.771499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.772118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5281.816323] Lustre: Skipped 1 previous similar message [ 5283.458291] Lustre: server umount lustre-OST0000 complete [ 5286.671148] LustreError: 168583:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5286.677314] Lustre: 167930:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5286.684827] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5286.695506] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5286.700607] LustreError: 167930:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001258ba5b x1872827567353600/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5286.708262] LustreError: 167930:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5286.712742] LustreError: 167930:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5286.717177] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5286.882430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.899122] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.914866] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5286.950479] Lustre: server umount lustre-MDT0000 complete [ 5289.944143] LustreError: 164833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067674 with bad export cookie 3474341084105571881 [ 5289.946609] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.955174] LustreError: 164833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5292.396392] Lustre: server umount lustre-MDT0001 complete [ 5306.385270] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5309.727416] Key type lgssc unregistered [ 5310.036676] LNet: 169339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5310.042664] LNetError: 169339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5310.072601] LNet: Removed LNI 192.168.201.124@tcp [ 5311.138848] Key type .llcrypt unregistered [ 5311.141035] Key type ._llcrypt unregistered [ 5335.064893] Key type ._llcrypt registered [ 5335.066240] Key type .llcrypt registered [ 5335.137713] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5335.778816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5335.787791] alg: No test for adler32 (adler32-zlib) [ 5336.812473] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5337.008099] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5338.632672] Key type lgssc registered [ 5339.941887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5348.312921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5348.353291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.089575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.553428] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5360.610464] LustreError: 170753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.649275] LustreError: 170753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5365.729074] LustreError: 170754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.544462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.870961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5372.433912] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5379.664596] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5385.490660] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5391.879513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5392.303674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.378617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5397.559975] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5404.997874] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5410.786401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5410.787498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5410.790477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.807189] Lustre: Skipped 1 previous similar message [ 5412.339192] Lustre: server umount lustre-OST0000 complete [ 5420.000570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5420.023874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.047364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.063823] Lustre: Skipped 1 previous similar message [ 5421.026311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.056931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.063073] Lustre: Skipped 1 previous similar message [ 5422.710718] Lustre: server umount lustre-MDT0000 complete [ 5426.146487] LustreError: 171812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5426.171721] LustreError: 171812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5426.866510] LustreError: 170733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067811 with bad export cookie 5913707330667355011 [ 5426.883348] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.286359] Lustre: server umount lustre-MDT0001 complete [ 5437.977783] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5441.160879] Key type lgssc unregistered [ 5441.572889] LNet: 174285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5441.583720] LNetError: 174285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5441.607172] LNet: Removed LNI 192.168.201.124@tcp [ 5442.522148] Key type .llcrypt unregistered [ 5442.528826] Key type ._llcrypt unregistered [ 5473.165270] Key type ._llcrypt registered [ 5473.166686] Key type .llcrypt registered [ 5473.261529] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5474.243562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5474.290347] alg: No test for adler32 (adler32-zlib) [ 5475.292815] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5475.521590] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5477.143360] Key type lgssc registered [ 5479.034728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5489.837130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5489.851714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5491.336547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.989769] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5500.901560] LustreError: 175987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.918732] LustreError: 175987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5506.032386] LustreError: 175986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.151937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.500855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5512.991914] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5522.963321] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5529.457588] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5536.726114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.498614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.020382] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5554.808432] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5561.145876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5571.555170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.585064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5576.676081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5576.685116] Lustre: Skipped 1 previous similar message [ 5577.592764] Lustre: server umount lustre-OST0000 complete [ 5581.792246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.808237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.811401] Lustre: Skipped 2 previous similar messages [ 5581.847293] Lustre: Skipped 1 previous similar message [ 5586.914248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5586.931918] Lustre: Skipped 1 previous similar message [ 5587.802628] Lustre: server umount lustre-MDT0000 complete [ 5592.014456] LustreError: 175967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067976 with bad export cookie 16948226426839512975 [ 5592.015924] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5592.037632] LustreError: 175967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5592.785240] Lustre: server umount lustre-MDT0001 complete [ 5606.234085] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5610.657434] Key type lgssc unregistered [ 5611.375700] LNet: 179553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5611.384564] LNetError: 179553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5611.405317] LNet: Removed LNI 192.168.201.124@tcp [ 5612.820659] Key type .llcrypt unregistered [ 5612.828964] Key type ._llcrypt unregistered [ 5633.959715] Key type ._llcrypt registered [ 5633.961699] Key type .llcrypt registered [ 5634.037241] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5635.192362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5635.371337] alg: No test for adler32 (adler32-zlib) [ 5636.532619] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5636.761048] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5638.543177] Key type lgssc registered [ 5639.853862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5655.378617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5655.428373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.405724] Lustre: *** cfs_fail_loc=135, val=0*** [ 5657.597150] LustreError: 181056:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5657.616498] LustreError: 181056:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -2 [ 5657.632287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5657.632287] [ 5657.645533] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5657.669674] LustreError: 181018:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5657.688378] LustreError: 181018:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5657.708167] LustreError: 181018:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5657.711059] LustreError: 181018:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5657.968336] Lustre: server umount lustre-MDT0000 complete [ 5657.970375] LustreError: 181018:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5667.148556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5667.544858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.543298] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5677.044463] LustreError: 181428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.071873] LustreError: 181428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5682.153883] LustreError: 181429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.268808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.730886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5690.161394] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5697.596380] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5702.408469] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.184635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.192832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5705.210821] Lustre: Skipped 1 previous similar message [ 5705.230804] Lustre: Skipped 1 previous similar message [ 5710.305645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.322959] Lustre: Skipped 1 previous similar message [ 5711.009637] Lustre: server umount lustre-MDT0000 complete [ 5711.337051] LustreError: 181424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.397922] LustreError: 181424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5715.928521] LustreError: 181410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068100 with bad export cookie 5471904371226949032 [ 5715.938474] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5716.151550] Lustre: server umount lustre-MDT0001 complete [ 5730.407363] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5732.775610] Key type lgssc unregistered [ 5733.082872] LNet: 183763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5733.086756] LNetError: 183763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5734.119206] LNet: Removed LNI 192.168.201.124@tcp [ 5735.062575] Key type .llcrypt unregistered [ 5735.071052] Key type ._llcrypt unregistered [ 5756.902279] Key type ._llcrypt registered [ 5756.904839] Key type .llcrypt registered [ 5757.008242] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5757.312484] Key type .llcrypt unregistered [ 5757.316208] Key type ._llcrypt unregistered [ 5772.839941] Key type ._llcrypt registered [ 5772.842471] Key type .llcrypt registered [ 5772.941325] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5773.945474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5774.006090] alg: No test for adler32 (adler32-zlib) [ 5775.136681] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5775.390202] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5777.103228] Key type lgssc registered [ 5778.350208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.133154] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.141931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5801.631452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5806.751547] LustreError: 185900:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 5833.375625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.576905] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5846.893034] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5853.755746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.850743] LustreError: 185236:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000deeda41c x1872828205893888/t0(0) o250->MGC192.168.201.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5854.225182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5854.316848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5859.314797] LustreError: 185925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.328825] LustreError: 185925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5859.611422] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5864.419608] LustreError: 185926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.370277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.514795] LustreError: 186938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.543177] LustreError: 186938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5868.956632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5872.993025] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5880.971435] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5888.346348] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5892.670112] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5904.865244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5904.869481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.875808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5904.895955] Lustre: Skipped 1 previous similar message [ 5904.910626] Lustre: Skipped 1 previous similar message [ 5909.990219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5910.786843] Lustre: server umount lustre-OST0000 complete [ 5915.104891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5915.116985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.131048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5915.131055] Lustre: Skipped 1 previous similar message [ 5915.182733] Lustre: Skipped 1 previous similar message [ 5920.226688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5920.229607] Lustre: Skipped 2 previous similar messages [ 5921.053622] Lustre: server umount lustre-MDT0000 complete [ 5924.681929] LustreError: 185908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068308 with bad export cookie 6184140665458154587 [ 5924.699230] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.708579] LustreError: 185908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5925.094272] Lustre: server umount lustre-MDT0001 complete [ 5936.975342] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 5939.906396] Key type lgssc unregistered [ 5940.197820] LNet: 189871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5940.217076] LNetError: 189871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5940.229242] LNet: Removed LNI 192.168.201.124@tcp [ 5941.358177] Key type .llcrypt unregistered [ 5941.360750] Key type ._llcrypt unregistered [ 5967.104859] Key type ._llcrypt registered [ 5967.108530] Key type .llcrypt registered [ 5967.234539] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 5968.525315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5968.682179] alg: No test for adler32 (adler32-zlib) [ 5969.910490] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5970.171477] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 5971.855174] Key type lgssc registered [ 5973.255651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5981.236827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5981.247342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.998711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.716062] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 5993.445660] LustreError: 191282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.487192] LustreError: 191282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5997.410650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.010929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.987611] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6012.496801] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6018.641982] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6025.261467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.613407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.723297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6032.344976] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6039.681557] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6044.562141] Lustre: Failing over lustre-MDT0000 [ 6045.124684] Lustre: server umount lustre-MDT0000 complete [ 6046.554313] LustreError: 191277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.235284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.237935] LustreError: 191283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.281031] Lustre: Skipped 1 previous similar message [ 6051.610219] LustreError: 191277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.634379] LustreError: 191277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6056.726671] LustreError: 191279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.763750] LustreError: 191279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6064.608078] Lustre: 190594:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786068432/real 1786068432] req@00000000605ce1b9 x1872828410388736/t0(0) o400->MGC192.168.201.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786068448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6064.659513] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6065.276127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.940422] LustreError: 191277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.962679] LustreError: 191277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6073.836759] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4b29533d1b526b5b [ 6073.860176] Lustre: MGC192.168.201.124@tcp: Connection restored to 0@lo (at 0@lo) [ 6074.257109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6074.399213] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6077.207917] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6079.479720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6079.509396] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6079.570839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6080.234781] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6087.754918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6093.469778] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6100.961958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6100.980435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.984246] LustreError: Skipped 1 previous similar message [ 6101.023871] Lustre: Skipped 1 previous similar message [ 6101.046633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6104.548866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.552389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.564535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6106.622930] Lustre: server umount lustre-OST0000 complete [ 6115.299542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6115.308604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.322853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6115.334872] Lustre: Skipped 1 previous similar message [ 6115.360611] Lustre: Skipped 3 previous similar messages [ 6116.555165] Lustre: server umount lustre-MDT0000 complete [ 6120.388812] LustreError: 191263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068504 with bad export cookie 5415951548806228827 [ 6120.398068] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.403440] LustreError: 191263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6121.001904] Lustre: server umount lustre-MDT0001 complete [ 6132.582283] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6134.949852] Key type lgssc unregistered [ 6135.251748] LNet: 196080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6135.256913] LNetError: 196080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6135.274863] LNet: Removed LNI 192.168.201.124@tcp [ 6136.086872] Key type .llcrypt unregistered [ 6136.089320] Key type ._llcrypt unregistered [ 6162.616725] Key type ._llcrypt registered [ 6162.619660] Key type .llcrypt registered [ 6162.728346] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6163.686501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6163.760089] alg: No test for adler32 (adler32-zlib) [ 6165.002941] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6165.384699] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6167.112101] Key type lgssc registered [ 6168.374296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6176.071947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6176.088528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.616158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.026909] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6186.979138] LustreError: 197493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.008686] LustreError: 197493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6192.105247] LustreError: 197494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.382532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.636255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6197.658758] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6205.585888] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6211.671168] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6220.024107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6220.254828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.313260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6227.596716] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6235.891674] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6238.466719] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6245.988629] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6268.892674] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6278.625326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6278.627078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.649801] LustreError: Skipped 1 previous similar message [ 6278.650972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.678740] Lustre: Skipped 1 previous similar message [ 6283.747711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.756865] Lustre: Skipped 2 previous similar messages [ 6284.722648] Lustre: server umount lustre-OST0000 complete [ 6288.864771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.866275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6288.899666] Lustre: Skipped 1 previous similar message [ 6288.917755] Lustre: Skipped 1 previous similar message [ 6293.989425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6294.006540] Lustre: Skipped 1 previous similar message [ 6294.731990] Lustre: server umount lustre-MDT0000 complete [ 6298.792956] LustreError: 197476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068683 with bad export cookie 8415880109862825933 [ 6298.797747] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.807088] LustreError: 197476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6299.419431] Lustre: server umount lustre-MDT0001 complete [ 6311.134639] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6314.124555] Key type lgssc unregistered [ 6314.455768] LNet: 201369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6314.465210] LNetError: 201369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6314.493326] LNet: Removed LNI 192.168.201.124@tcp [ 6315.857758] Key type .llcrypt unregistered [ 6315.860212] Key type ._llcrypt unregistered [ 6340.137308] Key type ._llcrypt registered [ 6340.139971] Key type .llcrypt registered [ 6340.233730] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6341.858220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6341.986649] alg: No test for adler32 (adler32-zlib) [ 6343.264809] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6343.666665] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6345.423731] Key type lgssc registered [ 6346.970578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6354.576735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6354.600874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.013440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.968419] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6366.177104] LustreError: 202637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.200480] LustreError: 202637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6370.347387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.780457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6375.964173] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6383.298460] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6389.683664] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6396.940106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.488510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.593773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6404.323792] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6412.182492] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.735318] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6427.180360] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6433.497852] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6441.668058] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6452.704758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6452.726349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.747616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.779105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.783326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.783989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.493587] Lustre: server umount lustre-OST0000 complete [ 6462.948204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6462.957939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.980278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6462.984751] Lustre: Skipped 1 previous similar message [ 6466.017870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6466.018382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.419640] Lustre: server umount lustre-MDT0000 complete [ 6471.137155] LustreError: 202631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.151894] LustreError: 202631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6471.911864] LustreError: 206263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068856 with bad export cookie 2936641690305603692 [ 6471.920220] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6471.925984] LustreError: 206263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6472.374574] Lustre: server umount lustre-MDT0001 complete [ 6483.973913] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6486.712303] Key type lgssc unregistered [ 6487.079749] LNet: 207468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6487.097980] LNetError: 207468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6488.168227] LNet: Removed LNI 192.168.201.124@tcp [ 6489.249407] Key type .llcrypt unregistered [ 6489.251677] Key type ._llcrypt unregistered [ 6515.355323] Key type ._llcrypt registered [ 6515.359455] Key type .llcrypt registered [ 6515.443194] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6516.360754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6516.458918] alg: No test for adler32 (adler32-zlib) [ 6517.597909] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6518.027033] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6519.784095] Key type lgssc registered [ 6521.135993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6530.910628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6530.970844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.726323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.135517] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6542.325879] LustreError: 208883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.350497] LustreError: 208883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6547.434262] LustreError: 208882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.370617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.667106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6555.224301] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6563.002149] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6569.512362] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.285288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.662070] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6577.793877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.301467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6585.387776] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6592.942043] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6608.870188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6608.877567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.901824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.892571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.920874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.929300] Lustre: Skipped 1 previous similar message [ 6613.840367] Lustre: server umount lustre-OST0000 complete [ 6620.135017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.139973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6620.146749] Lustre: Skipped 1 previous similar message [ 6620.163502] Lustre: Skipped 1 previous similar message [ 6624.199369] Lustre: server umount lustre-MDT0000 complete [ 6625.259333] LustreError: 209613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.277361] LustreError: 209613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6628.325142] LustreError: 211818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069012 with bad export cookie 2900014490997348095 [ 6628.333445] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6628.340556] LustreError: 211818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6628.801106] Lustre: server umount lustre-MDT0001 complete [ 6640.915774] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6644.478643] Key type lgssc unregistered [ 6644.783912] LNet: 212575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6644.789630] LNetError: 212575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6644.824793] LNet: Removed LNI 192.168.201.124@tcp [ 6645.863399] Key type .llcrypt unregistered [ 6645.865164] Key type ._llcrypt unregistered [ 6669.735704] Key type ._llcrypt registered [ 6669.740369] Key type .llcrypt registered [ 6669.907385] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6670.852829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6670.935843] alg: No test for adler32 (adler32-zlib) [ 6672.456706] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6672.828042] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6674.562709] Key type lgssc registered [ 6676.039280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6685.448419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6685.483619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.157579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.504809] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6697.445891] LustreError: 213842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.489813] LustreError: 213842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6701.983100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.596491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6708.635459] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6717.588760] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6725.174392] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6733.818070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.542429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.224488] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6750.158927] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.039699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6777.831402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.842345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6777.862663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.872686] Lustre: Skipped 1 previous similar message [ 6781.969893] Lustre: server umount lustre-OST0000 complete [ 6788.076541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.082931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6788.108237] Lustre: Skipped 2 previous similar messages [ 6788.126876] Lustre: Skipped 1 previous similar message [ 6792.919525] Lustre: server umount lustre-MDT0000 complete [ 6793.185306] LustreError: 214771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.201307] LustreError: 214771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6796.660764] LustreError: 213824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069180 with bad export cookie 12367574290248903514 [ 6796.669382] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6796.675489] LustreError: 213824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6797.172474] Lustre: server umount lustre-MDT0001 complete [ 6809.630279] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 6812.611412] Key type lgssc unregistered [ 6812.863918] LNet: 217554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6812.867545] LNetError: 217554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6812.889923] LNet: Removed LNI 192.168.201.124@tcp [ 6813.772458] Key type .llcrypt unregistered [ 6813.774528] Key type ._llcrypt unregistered [ 6833.804269] Key type ._llcrypt registered [ 6833.807074] Key type .llcrypt registered [ 6833.936624] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 6835.142070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6835.237397] alg: No test for adler32 (adler32-zlib) [ 6836.344393] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6836.575938] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 6838.271380] Key type lgssc registered [ 6839.365760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6847.683227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6847.697606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.353717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6854.795251] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6859.744423] LustreError: 218819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.752400] LustreError: 218819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6863.204781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.524186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6868.356747] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6876.400248] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6883.389796] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6890.692648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.140070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6895.299318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6897.654334] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6905.034660] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6914.690419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6914.799750] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6915.039861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6921.148626] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 6928.189772] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6929.759771] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6942.075251] Lustre: Permanently deactivating lustre-OST0001 [ 6942.096939] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6950.475735] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6961.477830] Lustre: Permanently reactivating lustre-OST0001 [ 6961.507925] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6961.515111] Lustre: Skipped 2 previous similar messages [ 6970.943541] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.957473] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6970.969557] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6970.983706] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6971.035727] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6974.965129] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.970786] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6974.978789] Lustre: Skipped 2 previous similar messages [ 6974.990334] Lustre: Skipped 1 previous similar message [ 6975.171814] Lustre: server umount lustre-OST0001 complete [ 6980.068294] LustreError: 221325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.069813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.071210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6980.082954] LustreError: 221325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6980.128886] Lustre: Skipped 1 previous similar message [ 6985.199274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6985.203613] Lustre: Skipped 1 previous similar message [ 6985.208442] LustreError: 221325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6985.226126] LustreError: 221325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6990.312640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.325315] LustreError: 220640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.326700] Lustre: Skipped 1 previous similar message [ 6990.354072] LustreError: 220640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6993.375457] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6993.532169] Lustre: server umount lustre-OST0000 complete [ 6996.960203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6996.978078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.994549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.666414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.669378] Lustre: Skipped 3 previous similar messages [ 7011.295769] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7011.544085] Lustre: server umount lustre-MDT0000 complete [ 7015.465124] LustreError: 218798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069399 with bad export cookie 7499461744199269463 [ 7015.480959] LustreError: 218798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7015.482554] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7015.796347] Lustre: server umount lustre-MDT0001 complete [ 7026.223506] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 7029.777536] Key type lgssc unregistered [ 7030.099961] LNet: 223920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7030.112680] LNetError: 223920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7030.137704] LNet: Removed LNI 192.168.201.124@tcp [ 7031.389735] Key type .llcrypt unregistered [ 7031.394028] Key type ._llcrypt unregistered [ 7076.456813] Key type ._llcrypt registered [ 7076.461255] Key type .llcrypt registered [ 7076.686905] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7078.234418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7078.268640] alg: No test for adler32 (adler32-zlib) [ 7079.433539] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7079.743815] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 7081.407187] Key type lgssc registered [ 7082.911091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7091.884543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7091.911492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.301902] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7093.365195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7093.565919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.411258] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7107.234896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.360810] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7107.390034] Lustre: 226129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7107.635457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7113.168864] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7122.196637] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7127.842046] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7135.703554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.122943] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7136.380799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.047880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7144.109847] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7151.786559] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7155.674497] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7163.635770] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7173.734376] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7180.610670] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7189.498572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7208.423213] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7208.433192] Lustre: Skipped 1 previous similar message [ 7245.612545] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7245.618442] Lustre: Skipped 4 previous similar messages [ 7310.048903] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7310.068562] Lustre: Skipped 7 previous similar messages [ 7338.465112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.473502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.501367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7339.004301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.007071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.027121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7344.098039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7344.111911] Lustre: Skipped 1 previous similar message [ 7349.218139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.223622] Lustre: Skipped 1 previous similar message [ 7352.287690] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7352.685130] Lustre: server umount lustre-OST0000 complete [ 7358.436847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7358.449256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.463922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.696714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.699356] Lustre: Skipped 4 previous similar messages [ 7371.231175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7371.435305] Lustre: server umount lustre-MDT0000 complete [ 7375.723488] LustreError: 226120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069760 with bad export cookie 4438082747023804304 [ 7375.733230] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7375.744991] LustreError: 226120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7376.462950] Lustre: server umount lustre-MDT0001 complete [ 7388.992592] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 7391.038286] Key type lgssc unregistered [ 7391.474803] LNet: 230928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7391.491423] LNetError: 230928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7391.515534] LNet: Removed LNI 192.168.201.124@tcp [ 7392.318281] Key type .llcrypt unregistered [ 7392.320730] Key type ._llcrypt unregistered [ 7421.558186] Key type ._llcrypt registered [ 7421.560693] Key type .llcrypt registered [ 7421.729603] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7423.834577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7423.932079] alg: No test for adler32 (adler32-zlib) [ 7425.140626] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7425.549622] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 7427.268108] Key type lgssc registered [ 7428.179661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7437.961927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7437.989614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.488872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.498660] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7449.082741] LustreError: 232344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.111056] LustreError: 232344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7454.178221] LustreError: 232343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7454.991768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.453641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7460.453582] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7469.676677] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7476.791034] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7485.347518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7485.652887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.325655] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 7502.057154] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7509.241951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7529.955901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.960748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.979016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7535.073720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7535.087893] Lustre: Skipped 1 previous similar message [ 7535.970707] Lustre: server umount lustre-OST0000 complete [ 7540.193341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.203072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7540.209425] Lustre: Skipped 2 previous similar messages [ 7540.236354] Lustre: Skipped 2 previous similar messages [ 7545.312377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.322941] Lustre: Skipped 1 previous similar message [ 7546.267753] Lustre: server umount lustre-MDT0000 complete [ 7550.443028] LustreError: 232344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.476943] LustreError: 232344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7550.525593] LustreError: 232323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069934 with bad export cookie 2401954738261883035 [ 7550.530776] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.538587] LustreError: 232323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7551.099568] Lustre: server umount lustre-MDT0001 complete [ 7564.729754] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 7568.207860] Key type lgssc unregistered [ 7568.481143] LNet: 236061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.509979] LNetError: 236061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7568.539654] LNet: Removed LNI 192.168.201.124@tcp [ 7569.714626] Key type .llcrypt unregistered [ 7569.717031] Key type ._llcrypt unregistered [ 7594.418556] Key type ._llcrypt registered [ 7594.420500] Key type .llcrypt registered [ 7594.553325] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7595.866690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7595.981712] alg: No test for adler32 (adler32-zlib) [ 7597.234442] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7597.538912] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 7599.231223] Key type lgssc registered [ 7600.439061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7870.804285] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 7873.995564] Key type lgssc unregistered [ 7874.252708] LNet: 238091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7874.270235] LNetError: 238091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7874.283649] LNet: Removed LNI 192.168.201.124@tcp [ 7875.026572] Key type .llcrypt unregistered [ 7875.030551] Key type ._llcrypt unregistered [ 7902.101365] Key type ._llcrypt registered [ 7902.103496] Key type .llcrypt registered [ 7902.176528] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 7924.210189] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 7925.823206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7925.881409] alg: No test for adler32 (adler32-zlib) [ 7927.025686] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7927.307844] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 7929.015312] Key type lgssc registered [ 7930.493722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7970.895334] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7696 sec ========== 22:39:12 (1786070352) [ 7972.576903] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:39:14 (1786070354) === [ 7976.208863] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:39:18 (1786070358) === [ 8023.369906] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 8025.791121] Key type lgssc unregistered [ 8026.191588] LNet: 243348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8026.205137] LNetError: 243348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8026.229225] LNet: Removed LNI 192.168.201.124@tcp [ 8027.340527] Key type .llcrypt unregistered [ 8027.346169] Key type ._llcrypt unregistered