[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 541288055 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003277] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.007839] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010182] LSM: Security Framework initializing [ 0.011089] Yama: becoming mindful. [ 0.012067] SELinux: Initializing. [ 0.014018] *** VALIDATE selinux *** [ 0.023314] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028143] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030155] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031156] *** VALIDATE tmpfs *** [ 0.033312] *** VALIDATE proc *** [ 0.034328] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036323] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043496] debug: unmapping init [mem 0xffffffffa4c59000-0xffffffffa4c60fff] [ 0.045224] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046871] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047038] ... version: 2 [ 0.048021] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050020] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.052969] ... event mask: 000000070000000f [ 0.053366] rcu: Hierarchical SRCU implementation. [ 0.055638] smp: Bringing up secondary CPUs ... [ 0.056681] x86: Booting SMP configuration: [ 0.057049] .... node #0, CPUs: #1 #2 #3 [ 0.061017] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155177] node 0 deferred pages initialised in 88ms [ 0.158102] devtmpfs: initialized [ 0.159318] x86/mm: Memory block size: 128MB [ 0.164214] gcov: version magic: 0x41383552 [ 0.167292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.171138] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173319] pinctrl core: initialized pinctrl subsystem [ 0.175177] [ 0.175651] ************************************************************* [ 0.178015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181028] ** ** [ 0.183016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.186024] ** ** [ 0.188027] ** This means that this kernel is built to expose internal ** [ 0.190018] ** IOMMU data structures, which may compromise security on ** [ 0.193020] ** your system. ** [ 0.195013] ** ** [ 0.198029] ** If you see this message and you are not debugging the ** [ 0.200020] ** kernel, report this immediately to your vendor! ** [ 0.202016] ** ** [ 0.205017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207020] ************************************************************* [ 0.210840] NET: Registered protocol family 16 [ 0.213567] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.223012] cpuidle: using governor menu [ 0.224672] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225573] PCI: Using configuration type 1 for base access [ 0.227158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236214] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244036] cryptd: max_cpu_qlen set to 1000 [ 0.246242] ACPI: Added _OSI(Module Device) [ 0.248023] ACPI: Added _OSI(Processor Device) [ 0.250023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256467] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.265466] ACPI: Interpreter enabled [ 0.266088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.268016] ACPI: Using IOAPIC for interrupt routing [ 0.269135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.273456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.283716] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.285069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.288032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.291108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.296438] acpiphp: Slot [2] registered [ 0.298151] acpiphp: Slot [5] registered [ 0.300161] acpiphp: Slot [6] registered [ 0.301172] acpiphp: Slot [7] registered [ 0.302139] acpiphp: Slot [8] registered [ 0.304134] acpiphp: Slot [9] registered [ 0.305079] acpiphp: Slot [10] registered [ 0.306117] acpiphp: Slot [3] registered [ 0.308122] acpiphp: Slot [4] registered [ 0.309095] acpiphp: Slot [11] registered [ 0.310167] acpiphp: Slot [12] registered [ 0.312148] acpiphp: Slot [13] registered [ 0.313126] acpiphp: Slot [14] registered [ 0.315109] acpiphp: Slot [15] registered [ 0.315925] acpiphp: Slot [16] registered [ 0.317133] acpiphp: Slot [17] registered [ 0.319176] acpiphp: Slot [18] registered [ 0.320122] acpiphp: Slot [19] registered [ 0.321135] acpiphp: Slot [20] registered [ 0.323120] acpiphp: Slot [21] registered [ 0.324089] acpiphp: Slot [22] registered [ 0.325103] acpiphp: Slot [23] registered [ 0.326045] acpiphp: Slot [24] registered [ 0.327106] acpiphp: Slot [25] registered [ 0.329122] acpiphp: Slot [26] registered [ 0.330136] acpiphp: Slot [27] registered [ 0.332133] acpiphp: Slot [28] registered [ 0.333106] acpiphp: Slot [29] registered [ 0.335247] acpiphp: Slot [30] registered [ 0.336106] acpiphp: Slot [31] registered [ 0.338113] PCI host bridge to bus 0000:00 [ 0.340031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.350027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357173] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373035] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377992] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.382025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388740] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.391900] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397713] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.416020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.422022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429373] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.442023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.449020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.488817] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522363] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550960] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.556022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.561015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.573017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.583463] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.595018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609035] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.646270] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.656029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.661026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.685955] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.688465] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690434] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.693458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.696342] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702036] iommu: Default domain type: Passthrough [ 0.704462] SCSI subsystem initialized [ 0.705158] ACPI: bus type USB registered [ 0.707097] usbcore: registered new interface driver usbfs [ 0.708080] usbcore: registered new interface driver hub [ 0.709100] usbcore: registered new device driver usb [ 0.711141] pps_core: LinuxPPS API ver. 1 registered [ 0.712013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715091] PTP clock support registered [ 0.718107] EDAC MC: Ver: 3.0.0 [ 0.720101] PCI: Using ACPI for IRQ routing [ 0.722107] NetLabel: Initializing [ 0.722979] NetLabel: domain hash size = 128 [ 0.724012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.726152] NetLabel: unlabeled traffic allowed by default [ 0.729188] vgaarb: loaded [ 0.731280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.732014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.736442] clocksource: Switched to clocksource kvm-clock [ 0.847654] VFS: Disk quotas dquot_6.6.0 [ 0.849280] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.851921] *** VALIDATE ramfs *** [ 0.853079] *** VALIDATE hugetlbfs *** [ 0.854291] pnp: PnP ACPI init [ 0.856312] pnp: PnP ACPI: found 6 devices [ 0.876666] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880120] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.882406] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.884637] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.886964] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.889476] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.892483] NET: Registered protocol family 2 [ 0.894481] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.900215] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904289] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910225] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914216] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917093] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.920434] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923715] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926472] NET: Registered protocol family 1 [ 0.928710] RPC: Registered named UNIX socket transport module. [ 0.930776] RPC: Registered udp transport module. [ 0.931859] RPC: Registered tcp transport module. [ 0.933392] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.935904] NET: Registered protocol family 44 [ 0.937466] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.939559] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.941425] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943814] PCI: CLS 0 bytes, default 64 [ 0.945608] Unpacking initramfs... [ 2.381730] debug: unmapping init [mem 0xffff8d6bbcc54000-0xffff8d6bbffbffff] [ 2.385883] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388325] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.391565] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.897443] Initialise system trusted keyrings [ 2.899379] Key type blacklist registered [ 2.901397] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.912416] zbud: loaded [ 2.915692] *** VALIDATE nfs *** [ 2.917036] *** VALIDATE nfs4 *** [ 2.918939] pstore: using deflate compression [ 2.923125] Platform Keyring initialized [ 3.045447] NET: Registered protocol family 38 [ 3.047634] Key type asymmetric registered [ 3.049437] Asymmetric key parser 'x509' registered [ 3.051755] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.055896] io scheduler mq-deadline registered [ 3.058261] io scheduler kyber registered [ 3.060242] io scheduler bfq registered [ 3.063186] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.066757] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.070351] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.073834] ACPI: Power Button [PWRF] [ 3.080155] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.086972] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.102224] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.109980] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.136389] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.165932] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.194262] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.199796] Non-volatile memory driver v1.3 [ 3.201532] Linux agpgart interface v0.103 [ 3.240429] virtio_blk virtio1: [vda] 147224 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.243368] vda: detected capacity change from 0 to 75378688 [ 3.256888] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.259531] vdb: detected capacity change from 0 to 1073741824 [ 3.274341] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.277289] vdc: detected capacity change from 0 to 2621440000 [ 3.291398] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.294184] vdd: detected capacity change from 0 to 2621440000 [ 3.316808] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.319901] vde: detected capacity change from 0 to 4294967296 [ 3.337649] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.340519] vdf: detected capacity change from 0 to 4294967296 [ 3.351187] libphy: Fixed MDIO Bus: probed [ 3.361076] usbcore: registered new interface driver usbserial_generic [ 3.363768] usbserial: USB Serial support registered for generic [ 3.366202] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.370530] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.372500] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.375051] mousedev: PS/2 mouse device common for all mice [ 3.378183] rtc_cmos 00:05: RTC can wake from S4 [ 3.381255] rtc_cmos 00:05: registered as rtc0 [ 3.381571] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.383082] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.389172] intel_pstate: CPU model not supported [ 3.391741] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.395831] hid: raw HID events driver (C) Jiri Kosina [ 3.399098] usbcore: registered new interface driver usbhid [ 3.399223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.401018] usbhid: USB HID core driver [ 3.401202] drop_monitor: Initializing network drop monitor service [ 3.409164] Initializing XFRM netlink socket [ 3.410638] NET: Registered protocol family 10 [ 3.415773] Segment Routing with IPv6 [ 3.417052] NET: Registered protocol family 17 [ 3.419380] mpls_gso: MPLS GSO support [ 3.425327] RAS: Correctable Errors collector initialized. [ 3.427042] AVX version of gcm_enc/dec engaged. [ 3.428282] AES CTR mode by8 optimization enabled [ 3.506454] sched_clock: Marking stable (3506410321, 0)->(4447295847, -940885526) [ 3.509108] registered taskstats version 1 [ 3.511288] Loading compiled-in X.509 certificates [ 3.512767] zswap: loaded using pool lzo/zbud [ 3.532575] Key type big_key registered [ 3.543256] Key type encrypted registered [ 3.544332] ima: No TPM chip found, activating TPM-bypass! [ 3.545502] ima: Allocated hash algorithm: sha1 [ 3.546462] ima: No architecture policies found [ 3.547433] evm: Initialising EVM extended attributes: [ 3.548457] evm: security.selinux [ 3.549164] evm: security.ima [ 3.549741] evm: security.capability [ 3.550408] evm: HMAC attrs: 0x1 [ 3.552180] rtc_cmos 00:05: setting system clock to 2026-09-11 11:39:16 UTC (1789126756) [ 3.556532] debug: unmapping init [mem 0xffffffffa5c03000-0xffffffffa5dfffff] [ 3.558458] debug: unmapping init [mem 0xffffffffa4982000-0xffffffffa4c58fff] [ 3.568092] Write protecting the kernel read-only data: 28672k [ 3.570445] debug: unmapping init [mem 0xffffffffa3003000-0xffffffffa31fffff] [ 3.572467] debug: unmapping init [mem 0xffffffffa3914000-0xffffffffa39fffff] [ 3.603563] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.610932] systemd[1]: Detected virtualization kvm. [ 3.612533] systemd[1]: Detected architecture x86-64. [ 3.614872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.639426] systemd[1]: No hostname configured. [ 3.641361] systemd[1]: Set hostname to . [ 3.643351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646114] systemd[1]: Initializing machine ID from random generator. [ 3.679392] random: ln: uninitialized urandom read (6 bytes read) [ 3.766514] random: systemd: uninitialized urandom read (16 bytes read) [ 3.769864] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.774256] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.782355] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.344214] device-mapper: uevent: version 1.0.3 [ 4.346804] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.034828] virtio_net virtio0 ens2: renamed from eth0 [ 5.045184] random: fast init done [ 5.115309] scsi host0: ata_piix [ 5.184225] scsi host1: ata_piix [ 5.185695] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.187420] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.712294] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.057055] random: crng init done [ 10.059202] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.395749] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.470417] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.694277] SELinux: Disabled at runtime. [ 11.755922] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.764256] systemd[1]: Detected virtualization kvm. [ 11.766334] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.162427] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.165128] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.169344] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.173269] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.176458] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.184143] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.188325] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK [[ 12.356704] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.597992] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.856308] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.886104] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.964223] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.976343] EDAC sbridge: Ver: 1.1.2 [ 14.530209] Key type dns_resolver registered [ 14.832352] NFS: Registering the id_resolver key type [ 14.834190] Key type id_resolver registered [ 14.835587] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-server login: [ 39.610693] libcfs: loading out-of-tree module taints kernel. [ 39.628628] Key type ._llcrypt registered [ 39.630180] Key type .llcrypt registered [ 39.690774] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 48.587369] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 49.369592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.375600] alg: No test for adler32 (adler32-zlib) [ 50.411434] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 50.748382] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 52.383202] Key type lgssc registered [ 53.114871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 54.433610] hrtimer: interrupt took 1659571 ns [ 68.010910] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.437438] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 129.634957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.694435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.081560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.155692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.326641] Lustre: lustre-MDT0000: new disk, initializing [ 131.516761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.551234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.755121] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.653630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.788864] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.839862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.849275] Lustre: Skipped 1 previous similar message [ 153.965583] Lustre: lustre-MDT0001: new disk, initializing [ 154.075933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.121556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.138857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.324769] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.161499] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.814666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.054816] Lustre: lustre-OST0000: new disk, initializing [ 177.058659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.063635] Lustre: 8419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.116372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.852103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 180.862820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 180.957431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 184.857877] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.471797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.654354] Lustre: lustre-OST0001: new disk, initializing [ 199.658064] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 199.674589] Lustre: 9490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.785609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.412406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 205.423138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 205.480924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.658972] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.997868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.004441] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.236824] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.152589] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing yml_node [ 245.942317] Lustre: DEBUG MARKER: Client: 2.17.57.137 [ 248.874257] Lustre: DEBUG MARKER: MDS: 2.17.57.137 [ 251.575608] Lustre: DEBUG MARKER: OSS: 2.17.57.137 [ 253.469754] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 07:43:23 EDT 2026 [ 272.303996] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 273.973723] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 292.322250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 292.337149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.358293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.926590] Lustre: server umount lustre-MDT0000 complete [ 300.993642] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127053 with bad export cookie 6897882472475003927 [ 301.006953] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 301.008940] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 303.078603] LustreError: 6488:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.083891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.086258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.086265] Lustre: Skipped 3 previous similar messages [ 303.096715] LustreError: 6488:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 303.142091] Lustre: Skipped 4 previous similar messages [ 305.121149] LustreError: 6487:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.131901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.143963] LustreError: 6487:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 305.176215] Lustre: Skipped 2 previous similar messages [ 310.241645] LustreError: 6488:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.244105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.264870] LustreError: 6488:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 310.278088] Lustre: Skipped 1 previous similar message [ 315.359158] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.937858] Lustre: server umount lustre-MDT0001 complete [ 320.797866] Lustre: server umount lustre-OST0000 complete [ 324.641715] Lustre: server umount lustre-OST0001 complete [ 333.983846] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 348.669470] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 388.836082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.263611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.302087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 389.400194] Lustre: lustre-MDT0000: new disk, initializing [ 389.500213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.542603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.331359] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 407.646878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.804920] Lustre: 15047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.869247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.871531] Lustre: Skipped 1 previous similar message [ 407.940821] Lustre: lustre-MDT0001: new disk, initializing [ 408.035820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.089849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.117509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.927531] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 424.633401] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 430.525925] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 437.892358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.120436] Lustre: lustre-OST0000: new disk, initializing [ 438.122827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 438.128884] Lustre: 17017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 438.250708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.675368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.717377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 439.878491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 445.001949] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 455.717899] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 460.985337] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.314304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 467.612824] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 467.816887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 470.501896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 470.512787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.550166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 470.552681] Lustre: Skipped 3 previous similar messages [ 476.746530] Lustre: server umount lustre-OST0000 complete [ 484.839656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 484.844303] LustreError: Skipped 1 previous similar message [ 484.852432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.868452] Lustre: Skipped 1 previous similar message [ 484.875346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.881753] Lustre: Skipped 2 previous similar messages [ 487.460637] Lustre: server umount lustre-MDT0000 complete [ 490.978975] LustreError: 15058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.004099] LustreError: 15058:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 491.578049] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127244 with bad export cookie 6897882472475005460 [ 491.588878] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.595034] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 491.952519] Lustre: server umount lustre-MDT0001 complete [ 495.796413] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:47:26 (1789127246) [ 502.694639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.284474] LustreError: 19282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.406637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.661321] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 514.024367] LustreError: 19283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.047324] LustreError: 19283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 517.010905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.560115] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 530.310590] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.306635] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 543.748960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.623613] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 558.256547] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 569.826500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 569.842911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.855584] Lustre: Skipped 1 previous similar message [ 569.865766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.879740] Lustre: Skipped 2 previous similar messages [ 575.302391] Lustre: server umount lustre-OST0000 complete [ 579.042944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.052491] LustreError: Skipped 1 previous similar message [ 579.060886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.074812] Lustre: Skipped 1 previous similar message [ 593.375773] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.484433] Lustre: server umount lustre-MDT0000 complete [ 597.045674] LustreError: 19262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127349 with bad export cookie 6897882472475006342 [ 597.060393] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.063246] LustreError: 19262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 597.380763] Lustre: server umount lustre-MDT0001 complete [ 608.869991] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 612.158051] Key type lgssc unregistered [ 612.581939] LNet: 22819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.600939] LNetError: 22819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.616930] LNet: Removed LNI 192.168.202.136@tcp [ 613.646191] Key type .llcrypt unregistered [ 613.648788] Key type ._llcrypt unregistered [ 636.677482] Key type ._llcrypt registered [ 636.682868] Key type .llcrypt registered [ 636.823147] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 637.917144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.930748] alg: No test for adler32 (adler32-zlib) [ 639.073857] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 639.400896] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 641.063501] Key type lgssc registered [ 642.012415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.830811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.861657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.391907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.039354] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 661.986459] LustreError: 24234:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.031822] LustreError: 24234:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 665.393640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.751714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 670.541053] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 678.634694] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 684.925971] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 691.834353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 692.216291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.314959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 698.418690] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 706.583909] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 725.575710] Lustre: server umount lustre-OST0000 complete [ 731.108738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.111000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.126090] Lustre: Skipped 1 previous similar message [ 731.148656] Lustre: Skipped 1 previous similar message [ 736.224781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.230211] Lustre: Skipped 1 previous similar message [ 740.450345] Lustre: server umount lustre-MDT0000 complete [ 741.343715] Lustre: 23544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789127478/real 1789127478] req@000000005794fca5 x1876036044224384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789127494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 741.344868] LustreError: 24234:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.351654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.378580] Lustre: 23544:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 741.392058] LustreError: 24234:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 744.688134] LustreError: 24215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127497 with bad export cookie 17962911329595013597 [ 744.703020] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.710765] LustreError: 24215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 745.159107] Lustre: server umount lustre-MDT0001 complete [ 756.188834] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 760.043712] Key type lgssc unregistered [ 760.304746] LNet: 28216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.315433] LNetError: 28216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.337206] LNet: Removed LNI 192.168.202.136@tcp [ 761.222338] Key type .llcrypt unregistered [ 761.224552] Key type ._llcrypt unregistered [ 786.632539] Key type ._llcrypt registered [ 786.635842] Key type .llcrypt registered [ 786.802992] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 788.569185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.605442] alg: No test for adler32 (adler32-zlib) [ 789.691725] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 790.087748] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 791.856402] Key type lgssc registered [ 793.376970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.260703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.277512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.824070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.896898] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 813.042728] LustreError: 29631:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.068575] LustreError: 29631:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 817.498400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.969851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 822.743978] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 830.386898] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 836.312634] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 850.390436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 850.737682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.538865] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 866.225466] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 872.271435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 878.561366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 878.574120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.605676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.659037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 882.662937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.663080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.697256] Lustre: Skipped 1 previous similar message [ 887.778136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 891.871514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.088290] Lustre: server umount lustre-OST0000 complete [ 898.018016] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.020358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.051711] Lustre: Skipped 1 previous similar message [ 898.083267] Lustre: Skipped 2 previous similar messages [ 903.137824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.145565] Lustre: Skipped 1 previous similar message [ 911.327155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 911.554868] Lustre: server umount lustre-MDT0000 complete [ 913.378644] LustreError: 30358:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.390672] LustreError: 30358:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 915.091406] LustreError: 30365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127668 with bad export cookie 16538332191961459419 [ 915.100894] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 915.105139] LustreError: 30365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 915.423731] Lustre: server umount lustre-MDT0001 complete [ 928.849717] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 932.420992] Key type lgssc unregistered [ 932.856449] LNet: 33516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.860040] LNetError: 33516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.885985] LNet: Removed LNI 192.168.202.136@tcp [ 934.022576] Key type .llcrypt unregistered [ 934.026368] Key type ._llcrypt unregistered [ 961.992952] Key type ._llcrypt registered [ 961.994803] Key type .llcrypt registered [ 962.143340] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 963.267462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.335603] alg: No test for adler32 (adler32-zlib) [ 964.607663] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 964.818673] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 966.519196] Key type lgssc registered [ 967.929658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.618885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.656815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.290919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.860608] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 988.649961] LustreError: 34948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.690209] LustreError: 34948:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 993.766306] LustreError: 34947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.901146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.298728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 999.731513] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1008.853871] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1015.786762] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1024.627952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1024.963474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.000692] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1040.388374] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1048.558464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1063.921672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.924815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.945874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.958478] Lustre: Skipped 1 previous similar message [ 1063.981545] Lustre: Skipped 1 previous similar message [ 1067.357689] Lustre: server umount lustre-OST0000 complete [ 1074.145132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.146305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.169045] Lustre: Skipped 1 previous similar message [ 1074.175180] Lustre: Skipped 1 previous similar message [ 1078.246760] Lustre: server umount lustre-MDT0000 complete [ 1079.264839] LustreError: 34947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.295591] LustreError: 34947:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1081.888631] LustreError: 34928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127834 with bad export cookie 17184752652677992156 [ 1081.893795] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.906700] LustreError: 34928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1082.363887] Lustre: server umount lustre-MDT0001 complete [ 1093.409617] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1096.385346] Key type lgssc unregistered [ 1096.703961] LNet: 38508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.711151] LNetError: 38508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.741092] LNet: Removed LNI 192.168.202.136@tcp [ 1097.655685] Key type .llcrypt unregistered [ 1097.659541] Key type ._llcrypt unregistered [ 1121.335216] Key type ._llcrypt registered [ 1121.338664] Key type .llcrypt registered [ 1121.482924] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1122.360710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1122.401679] alg: No test for adler32 (adler32-zlib) [ 1123.540476] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1123.837903] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1125.543202] Key type lgssc registered [ 1126.784032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1135.199256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1135.213272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.958129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.264185] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1146.345366] LustreError: 39924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.366414] LustreError: 39924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1150.778404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.250257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.114265] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1163.704786] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1169.951098] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1176.925597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.246542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.363163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1183.848798] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1191.759794] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1202.160246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1202.160443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.173698] LustreError: Skipped 1 previous similar message [ 1202.193323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.204880] Lustre: Skipped 1 previous similar message [ 1203.804783] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1206.810279] Lustre: server umount lustre-OST0000 complete [ 1228.258821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1228.280048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.300837] Lustre: Skipped 1 previous similar message [ 1228.324456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.864892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.881914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.985636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.005320] Lustre: Skipped 2 previous similar messages [ 1242.079194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1242.346303] Lustre: server umount lustre-MDT0000 complete [ 1246.626204] LustreError: 42626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127999 with bad export cookie 13972654548100556285 [ 1246.631206] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1246.638617] LustreError: 42626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1253.055311] Lustre: server umount lustre-MDT0001 complete [ 1264.805550] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1268.061710] Key type lgssc unregistered [ 1268.370808] LNet: 43635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.375952] LNetError: 43635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.413854] LNet: Removed LNI 192.168.202.136@tcp [ 1270.369893] Key type .llcrypt unregistered [ 1270.376144] Key type ._llcrypt unregistered [ 1295.312472] Key type ._llcrypt registered [ 1295.314490] Key type .llcrypt registered [ 1295.474551] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1296.949524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.187410] alg: No test for adler32 (adler32-zlib) [ 1298.322399] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1298.593742] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1300.385311] Key type lgssc registered [ 1302.072581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1313.456506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1313.520661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.443473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.372567] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1325.549341] LustreError: 45059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.566646] LustreError: 45059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1329.919914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.440948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.512662] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1343.886368] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1351.319356] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.238857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.610151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.706286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1366.112672] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1374.266441] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1386.059499] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1386.466899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.493760] Lustre: Skipped 1 previous similar message [ 1389.796725] Lustre: server umount lustre-MDT0000 complete [ 1390.066270] LustreError: 46901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.586638] LustreError: 45788:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.911953] LustreError: 45042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128146 with bad export cookie 12811780532566854824 [ 1393.913369] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.932296] LustreError: 45042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1395.263725] LustreError: 45056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.287054] LustreError: 45056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1395.314661] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [ 1395.321562] Lustre: Skipped 3 previous similar messages [ 1396.705801] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.717978] Lustre: Skipped 1 previous similar message [ 1396.727979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1398.754641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1400.370679] LustreError: 45054:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.392789] LustreError: 45054:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1400.513800] Lustre: server umount lustre-MDT0001 complete [ 1421.133472] Lustre: server umount lustre-OST0000 complete [ 1438.289295] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1441.600436] Key type lgssc unregistered [ 1441.967275] LNet: 48924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.983319] LNetError: 48924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.011589] LNet: Removed LNI 192.168.202.136@tcp [ 1442.969792] Key type .llcrypt unregistered [ 1442.976261] Key type ._llcrypt unregistered [ 1470.092885] Key type ._llcrypt registered [ 1470.095400] Key type .llcrypt registered [ 1470.182811] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1471.433856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.513508] alg: No test for adler32 (adler32-zlib) [ 1472.642640] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1472.992759] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1474.951186] Key type lgssc registered [ 1476.586981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1485.570090] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1485.591665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.153529] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1506.274569] LustreError: 50312:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 1532.959279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.479845] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1550.610033] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1717.545785] Lustre: server umount lustre-OST0000 complete [ 1737.490812] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1740.206699] Key type lgssc unregistered [ 1740.535922] LNet: 51994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.547209] LNetError: 51994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.584074] LNet: Removed LNI 192.168.202.136@tcp [ 1741.533873] Key type .llcrypt unregistered [ 1741.542430] Key type ._llcrypt unregistered [ 1765.153617] Key type ._llcrypt registered [ 1765.155574] Key type .llcrypt registered [ 1765.273621] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1766.398964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1766.653260] alg: No test for adler32 (adler32-zlib) [ 1767.865513] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1768.219325] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1769.921216] Key type lgssc registered [ 1771.172496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1780.120783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1780.183273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.668803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.632762] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1791.973049] LustreError: 53409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.999391] LustreError: 53409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1796.099281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.570744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.684980] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1808.090397] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1814.070352] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1820.573128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.001872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.182442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1828.317230] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1837.312679] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1841.643638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.653081] Lustre: Skipped 1 previous similar message [ 1841.654079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.179261] Lustre: server umount lustre-OST0000 complete [ 1851.876705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.889253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.901720] Lustre: Skipped 1 previous similar message [ 1851.916694] Lustre: Skipped 2 previous similar messages [ 1856.977735] Lustre: server umount lustre-MDT0000 complete [ 1856.997214] LustreError: 53405:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.016634] LustreError: 53405:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1860.792503] LustreError: 53389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128613 with bad export cookie 15054108051735327137 [ 1860.803880] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1860.810282] LustreError: 53389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1861.402546] Lustre: server umount lustre-MDT0001 complete [ 1872.274679] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1874.773866] Key type lgssc unregistered [ 1875.007655] LNet: 56942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.020132] LNetError: 56942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.070841] LNet: Removed LNI 192.168.202.136@tcp [ 1876.745388] Key type .llcrypt unregistered [ 1876.748146] Key type ._llcrypt unregistered [ 1901.560983] Key type ._llcrypt registered [ 1901.565929] Key type .llcrypt registered [ 1901.690963] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1902.853702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.904258] alg: No test for adler32 (adler32-zlib) [ 1904.202765] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1904.529819] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1906.255504] Key type lgssc registered [ 1907.371176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1915.390774] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1915.405346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1930.911403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1936.032051] LustreError: 58337:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 1962.783952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.513378] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1978.213386] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1985.414585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.142490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.257100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1991.139293] LustreError: 58363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.177507] LustreError: 58363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1991.766327] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1996.258941] LustreError: 58362:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.285887] LustreError: 59373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.295174] LustreError: 59373:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2000.374918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.671244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2005.448059] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2014.449435] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.604450] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2026.464107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.473598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.499842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.516314] Lustre: Skipped 1 previous similar message [ 2026.539351] Lustre: Skipped 1 previous similar message [ 2030.696623] Lustre: server umount lustre-OST0000 complete [ 2057.188696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.209678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2057.216761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.851841] Lustre: server umount lustre-MDT0000 complete [ 2062.304582] LustreError: 59378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.334360] LustreError: 59378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2066.413861] LustreError: 61167:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.438424] LustreError: 61167:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2066.714651] LustreError: 58343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128819 with bad export cookie 15920524522131457666 [ 2066.722079] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.733181] LustreError: 58343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2081.247837] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2081.597740] Lustre: server umount lustre-MDT0001 complete [ 2096.345409] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2099.647345] Key type lgssc unregistered [ 2099.968706] LNet: 62094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.982817] LNetError: 62094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.019693] LNet: Removed LNI 192.168.202.136@tcp [ 2101.338221] Key type .llcrypt unregistered [ 2101.340314] Key type ._llcrypt unregistered [ 2127.392431] Key type ._llcrypt registered [ 2127.396453] Key type .llcrypt registered [ 2127.513783] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2128.518619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2128.720122] alg: No test for adler32 (adler32-zlib) [ 2129.927097] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2130.286108] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2131.983708] Key type lgssc registered [ 2133.160697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2141.829224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2141.844600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.315696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.620787] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2153.449717] LustreError: 63506:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.461539] LustreError: 63506:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2158.561298] LustreError: 63507:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.673124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.184744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2164.687609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2172.775356] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2178.991336] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2186.102045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2186.443525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.542898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2193.066306] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2201.415985] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2219.488338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2219.496248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.507749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.522150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.528617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.528804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2225.670390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2225.677039] Lustre: Skipped 2 previous similar messages [ 2230.762377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2233.316803] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2233.703865] Lustre: server umount lustre-OST0000 complete [ 2240.997941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2241.009771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.020102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.046738] Lustre: Skipped 1 previous similar message [ 2241.059890] Lustre: Skipped 2 previous similar messages [ 2245.658975] Lustre: server umount lustre-MDT0000 complete [ 2246.114313] LustreError: 63506:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.138889] LustreError: 63506:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2250.110071] LustreError: 63487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129003 with bad export cookie 3478947752506172023 [ 2250.117818] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.127651] LustreError: 63487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2250.446368] Lustre: server umount lustre-MDT0001 complete [ 2262.721405] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2265.513653] Key type lgssc unregistered [ 2265.808732] LNet: 67071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.822858] LNetError: 67071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.842166] LNet: Removed LNI 192.168.202.136@tcp [ 2266.826493] Key type .llcrypt unregistered [ 2266.836399] Key type ._llcrypt unregistered [ 2295.131708] Key type ._llcrypt registered [ 2295.137475] Key type .llcrypt registered [ 2295.219631] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2296.176155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2296.195388] alg: No test for adler32 (adler32-zlib) [ 2297.255531] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2297.472151] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2299.143267] Key type lgssc registered [ 2300.690221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2310.827961] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 08:17:40 (1789129060) [ 2318.345563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2318.360745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.869192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.930559] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2329.062612] LustreError: 68775:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.080078] LustreError: 68775:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2334.185396] LustreError: 68776:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.629225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.013121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2340.959672] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2350.172286] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2356.977707] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2364.781891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2365.447688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.425214] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2381.597568] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2390.088762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2400.248780] Lustre: Failing over lustre-MDT0000 [ 2400.629472] Lustre: server umount lustre-MDT0000 complete [ 2401.765136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2401.771210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.781179] LustreError: 68775:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.441594] LustreError: 71268:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.352601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.531036] LustreError: 68771:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.558297] LustreError: 68771:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2411.354591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.536454] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.753209] Lustre: *** cfs_fail_loc=135, val=0*** [ 2411.878374] LustreError: 71888:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2411.882721] LustreError: 71888:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -2 [ 2411.905500] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2411.905500] [ 2411.915643] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2411.936327] LustreError: 71877:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2411.947266] LustreError: 71877:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2411.956170] LustreError: 71877:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2411.963976] LustreError: 71877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2411.997371] Lustre: server umount lustre-MDT0000 complete [ 2411.999645] LustreError: 71877:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2419.291067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.364607] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.376952] LustreError: 68767:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2419.597587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.657340] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2419.766797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2424.835437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2424.884422] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2424.945811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2425.067451] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2435.051671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.057972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.062839] Lustre: Skipped 2 previous similar messages [ 2435.082121] Lustre: Skipped 1 previous similar message [ 2439.853987] Lustre: server umount lustre-OST0000 complete [ 2445.282979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.286082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.310599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.331907] Lustre: Skipped 1 previous similar message [ 2450.406838] LustreError: 68772:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.434060] LustreError: 68772:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 2450.552478] Lustre: server umount lustre-MDT0000 complete [ 2454.161837] LustreError: 68756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129207 with bad export cookie 115168250302666929 [ 2454.165066] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.171437] LustreError: 68756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2454.510032] Lustre: server umount lustre-MDT0001 complete [ 2467.450546] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2470.558506] Key type lgssc unregistered [ 2470.856694] LNet: 73670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.861858] LNetError: 73670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.889029] LNet: Removed LNI 192.168.202.136@tcp [ 2472.018535] Key type .llcrypt unregistered [ 2472.019708] Key type ._llcrypt unregistered [ 2502.575561] Key type ._llcrypt registered [ 2502.587500] Key type .llcrypt registered [ 2502.702417] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2503.922050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.969235] alg: No test for adler32 (adler32-zlib) [ 2504.937849] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2505.053959] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2506.648703] Key type lgssc registered [ 2508.208586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2517.061470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2517.081427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.688910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.569508] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2528.227464] LustreError: 75083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.251946] LustreError: 75083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2533.356251] LustreError: 75082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.341639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.854774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2539.689996] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2548.596250] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2557.284830] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2565.981893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2566.338905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.925624] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2581.516475] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2588.857224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2600.050299] Lustre: Failing over lustre-MDT0000 [ 2600.469942] Lustre: server umount lustre-MDT0000 complete [ 2601.445900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.458833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.478163] LustreError: 76082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.589053] LustreError: 75077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.524632] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.538920] Lustre: Skipped 1 previous similar message [ 2608.693196] LustreError: 77527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.725971] LustreError: 77527:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2611.278947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.432420] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.585773] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2611.747419] LustreError: 78195:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2611.755285] LustreError: 78195:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -9 [ 2611.770042] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2611.770042] [ 2611.784267] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2611.799680] LustreError: 78183:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2611.813978] LustreError: 78183:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2611.834081] LustreError: 78183:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2611.837896] LustreError: 78183:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2611.889130] Lustre: server umount lustre-MDT0000 complete [ 2611.892954] LustreError: 78183:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2618.931288] LustreError: 75077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.973275] LustreError: 75077:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2619.702420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.944450] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.952219] LustreError: 78542:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.136@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f45917cb) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2619.986757] LustreError: 75074:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2620.365214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.435644] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2624.060862] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2625.524543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2625.575183] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2625.629742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2625.698954] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2635.749355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.753487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.761488] Lustre: Skipped 1 previous similar message [ 2635.774504] Lustre: Skipped 1 previous similar message [ 2640.218788] Lustre: server umount lustre-OST0000 complete [ 2645.997285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2645.999414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.033779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2646.046545] Lustre: Skipped 1 previous similar message [ 2651.106900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.114690] Lustre: Skipped 1 previous similar message [ 2655.206666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.224699] Lustre: Skipped 1 previous similar message [ 2659.295693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2659.508714] Lustre: server umount lustre-MDT0000 complete [ 2660.323508] LustreError: 75083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.342824] LustreError: 75083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2663.143923] LustreError: 77687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129416 with bad export cookie 15460185155824445307 [ 2663.166193] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2663.179016] LustreError: 77687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2663.471682] Lustre: server umount lustre-MDT0001 complete [ 2674.659465] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2677.133611] Key type lgssc unregistered [ 2677.458726] LNet: 79979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.468869] LNetError: 79979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.495219] LNet: Removed LNI 192.168.202.136@tcp [ 2678.344836] Key type .llcrypt unregistered [ 2678.346870] Key type ._llcrypt unregistered [ 2694.055519] Key type ._llcrypt registered [ 2694.058357] Key type .llcrypt registered [ 2694.151197] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2694.473489] Key type .llcrypt unregistered [ 2694.475043] Key type ._llcrypt unregistered [ 2721.196749] Key type ._llcrypt registered [ 2721.203760] Key type .llcrypt registered [ 2721.320681] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2722.382773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2722.622409] alg: No test for adler32 (adler32-zlib) [ 2723.637848] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2723.866141] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2725.695166] Key type lgssc registered [ 2727.133974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2735.248460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2735.276783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.708645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.745122] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2746.851214] LustreError: 81887:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.877032] LustreError: 81887:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2750.874650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.258308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2755.874946] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2764.044250] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2769.469900] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2775.528529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.841133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.278421] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2782.008515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2789.500467] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2802.665330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.669956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.676085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.682662] Lustre: Skipped 1 previous similar message [ 2802.707481] LustreError: Skipped 1 previous similar message [ 2805.917372] Lustre: server umount lustre-OST0000 complete [ 2812.896730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2812.897324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.903300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.903310] Lustre: Skipped 1 previous similar message [ 2812.918021] Lustre: Skipped 1 previous similar message [ 2818.017158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.036823] Lustre: Skipped 2 previous similar messages [ 2822.127886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.156351] Lustre: Skipped 1 previous similar message [ 2824.159332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2824.662641] Lustre: server umount lustre-MDT0000 complete [ 2827.234497] LustreError: 81887:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.248086] LustreError: 81887:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2829.030695] LustreError: 81868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129581 with bad export cookie 16501843261245132030 [ 2829.037106] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2829.040872] LustreError: 81868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2829.226395] Lustre: server umount lustre-MDT0001 complete [ 2840.364227] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2843.200980] Key type lgssc unregistered [ 2843.571865] LNet: 85424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2843.590117] LNetError: 85424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2843.605851] LNet: Removed LNI 192.168.202.136@tcp [ 2844.452571] Key type .llcrypt unregistered [ 2844.455690] Key type ._llcrypt unregistered [ 2866.014350] Key type ._llcrypt registered [ 2866.016508] Key type .llcrypt registered [ 2866.115174] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2867.109293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2867.290659] alg: No test for adler32 (adler32-zlib) [ 2868.356287] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2868.604889] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2870.329572] Key type lgssc registered [ 2871.604539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.869920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2879.897210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.399526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.747412] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2891.750804] LustreError: 86817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.782445] LustreError: 86817:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2895.497211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.836575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.178721] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2910.835608] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2917.835343] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2927.312335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.720678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.636769] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2942.694806] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2951.292571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2961.890212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.908325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.910525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.921910] Lustre: Skipped 1 previous similar message [ 2961.956889] Lustre: Skipped 1 previous similar message [ 2963.201507] Lustre: server umount lustre-OST0000 complete [ 2967.520257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2967.529132] LustreError: Skipped 1 previous similar message [ 2967.533137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.552908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.130732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.132135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.804285] Lustre: server umount lustre-MDT0000 complete [ 2977.266513] LustreError: 87971:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.327150] LustreError: 87971:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2978.347674] LustreError: 86797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129731 with bad export cookie 6939346384389848624 [ 2978.357797] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.368651] LustreError: 86797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2979.010339] Lustre: server umount lustre-MDT0001 complete [ 2993.845157] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2998.579184] Key type lgssc unregistered [ 2998.848635] LNet: 90375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2998.854705] LNetError: 90375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2998.866912] LNet: Removed LNI 192.168.202.136@tcp [ 2999.672660] Key type .llcrypt unregistered [ 2999.674612] Key type ._llcrypt unregistered [ 3029.338219] Key type ._llcrypt registered [ 3029.340562] Key type .llcrypt registered [ 3029.422720] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3030.949644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.056192] alg: No test for adler32 (adler32-zlib) [ 3032.371893] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3032.645714] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3034.305066] Key type lgssc registered [ 3035.685870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.649567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3043.675589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.130693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.240830] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3055.585707] LustreError: 91794:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.593180] LustreError: 91794:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3060.713042] LustreError: 91793:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.930932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.597531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.574844] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3075.958924] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3082.033475] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3088.717139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3089.100110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.164989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3095.888659] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3104.233653] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3119.085961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3119.104825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.128493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.151483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.176561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3122.185692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.141491] Lustre: server umount lustre-OST0000 complete [ 3132.392747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.417180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.434644] Lustre: Skipped 1 previous similar message [ 3132.459935] Lustre: Skipped 1 previous similar message [ 3137.160824] Lustre: server umount lustre-MDT0000 complete [ 3137.505713] LustreError: 91794:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.528994] LustreError: 91794:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3141.816148] LustreError: 94289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129894 with bad export cookie 11027988167358630879 [ 3141.821681] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.827278] LustreError: 94289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3142.462779] Lustre: server umount lustre-MDT0001 complete [ 3154.994257] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3158.468732] Key type lgssc unregistered [ 3158.828421] LNet: 95357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3158.838711] LNetError: 95357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3158.855468] LNet: Removed LNI 192.168.202.136@tcp [ 3159.881480] Key type .llcrypt unregistered [ 3159.884270] Key type ._llcrypt unregistered [ 3189.951752] Key type ._llcrypt registered [ 3189.954540] Key type .llcrypt registered [ 3190.139712] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3192.599322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3192.660702] alg: No test for adler32 (adler32-zlib) [ 3193.869849] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3194.399332] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3196.143351] Key type lgssc registered [ 3198.178859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3207.464291] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3207.479635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.071796] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3228.191572] LustreError: 96745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 3254.879321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.426148] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3267.533414] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3274.998653] Lustre: server umount lustre-OST0000 complete [ 3289.106940] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:33:58 (1789130038) [ 3297.747385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.311527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.400595] LustreError: 98385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.418072] LustreError: 98385:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3305.612507] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3308.519923] LustreError: 98386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.638714] LustreError: 98385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.038369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.467800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3320.965996] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3329.604620] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3336.837272] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3344.574336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.034299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.712843] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3366.997822] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3368.709817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3386.336870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.341521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.370741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.415030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.416815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.424523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3390.629560] Lustre: server umount lustre-OST0000 complete [ 3397.602606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3397.614552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.633458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.648689] Lustre: Skipped 1 previous similar message [ 3399.656439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.673869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.239112] Lustre: server umount lustre-MDT0000 complete [ 3404.770166] LustreError: 98385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.795993] LustreError: 98385:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3406.332648] LustreError: 100230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130159 with bad export cookie 13133375204523596452 [ 3406.350981] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.358154] LustreError: 100230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3407.111340] Lustre: server umount lustre-MDT0001 complete [ 3421.709723] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3425.410289] Key type lgssc unregistered [ 3425.605595] LNet: 102084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3425.608241] LNetError: 102084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3425.616013] LNet: Removed LNI 192.168.202.136@tcp [ 3426.541548] Key type .llcrypt unregistered [ 3426.543626] Key type ._llcrypt unregistered [ 3477.031248] Key type ._llcrypt registered [ 3477.038706] Key type .llcrypt registered [ 3477.367727] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3480.042811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.085531] alg: No test for adler32 (adler32-zlib) [ 3481.159602] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3481.382797] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3483.079194] Key type lgssc registered [ 3485.115449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3495.579474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3495.601080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.423966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.931116] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3507.680927] LustreError: 104274:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.715777] LustreError: 104274:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3512.804495] LustreError: 104275:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.564557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.057810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3521.410844] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3531.848799] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3539.254340] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3548.846139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3549.558691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.252125] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3567.847852] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3568.069421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3578.342327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.342732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.361766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.383626] Lustre: Skipped 1 previous similar message [ 3583.259595] Lustre: server umount lustre-OST0000 complete [ 3588.576344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.597360] Lustre: Skipped 1 previous similar message [ 3588.603104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.185236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3593.206125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.261502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.037656] Lustre: server umount lustre-MDT0000 complete [ 3598.820270] LustreError: 104270:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.838275] LustreError: 104270:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3600.755498] LustreError: 104254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130353 with bad export cookie 821445407849006064 [ 3600.770857] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.783861] LustreError: 104254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3601.371551] Lustre: server umount lustre-MDT0001 complete [ 3613.440984] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3616.634904] Key type lgssc unregistered [ 3616.822643] LNet: 107813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3616.826886] LNetError: 107813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3616.837411] LNet: Removed LNI 192.168.202.136@tcp [ 3617.400421] Key type .llcrypt unregistered [ 3617.403421] Key type ._llcrypt unregistered [ 3636.984523] Key type ._llcrypt registered [ 3636.987220] Key type .llcrypt registered [ 3637.135302] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3638.694682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3638.732777] alg: No test for adler32 (adler32-zlib) [ 3639.820144] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3640.092708] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3641.743856] Key type lgssc registered [ 3643.042082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3651.239619] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3651.262488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.911438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3672.038575] LustreError: 109058:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 3698.720164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.582927] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3718.587388] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3728.302254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.658021] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3728.665911] LustreError: 110085:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3728.670964] LustreError: 110085:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3728.678306] LustreError: 110085:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3728.809626] Lustre: server umount lustre-MDT0000 complete [ 3728.812486] LustreError: 110085:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3749.279141] Lustre: 108394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130486/real 1789130486] req@00000000f1e424b1 x1876039190973824/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789130502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3749.320209] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.332084] LustreError: 109070:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3769.823621] LustreError: 109070:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3770.047392] Lustre: server umount lustre-OST0000 complete [ 3781.633147] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3794.936901] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3840.771452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.179551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3841.230684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3841.320647] Lustre: lustre-MDT0000: new disk, initializing [ 3841.387166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.407494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3846.786122] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3858.728383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.877459] Lustre: 112883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3858.938780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3858.945531] Lustre: Skipped 1 previous similar message [ 3859.065711] Lustre: lustre-MDT0001: new disk, initializing [ 3859.193940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.228724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3859.241843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3863.984775] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3874.864623] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3881.029715] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3888.412490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.776872] Lustre: lustre-OST0000: new disk, initializing [ 3888.780266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3888.785614] Lustre: 114854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3888.904557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.895450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3894.904828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3895.038553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3896.460612] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3907.428995] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3913.412262] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3913.768513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3919.781573] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3920.110680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3925.984627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3925.987444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.007090] LustreError: Skipped 1 previous similar message [ 3926.009767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.043497] Lustre: Skipped 1 previous similar message [ 3928.601337] Lustre: server umount lustre-OST0000 complete [ 3936.228943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3936.234491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.260246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.280402] Lustre: Skipped 1 previous similar message [ 3936.301796] Lustre: Skipped 2 previous similar messages [ 3939.358631] Lustre: server umount lustre-MDT0000 complete [ 3943.675335] LustreError: 112876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130696 with bad export cookie 13206848960729392622 [ 3943.688529] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3944.038200] Lustre: server umount lustre-MDT0001 complete [ 3955.054461] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 08:45:05 (1789130705) [ 3959.196542] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3979.614449] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3989.302267] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4028.142410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.435958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4028.458461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4028.550934] Lustre: lustre-MDT0000: new disk, initializing [ 4028.632033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.657634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4032.961798] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4044.136744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.394087] Lustre: 119757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4044.496820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4044.500973] Lustre: Skipped 1 previous similar message [ 4044.615103] Lustre: lustre-MDT0001: new disk, initializing [ 4044.776162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4044.825288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4044.838279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4050.210886] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4060.575666] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.628458] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.797340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.273436] Lustre: lustre-OST0000: new disk, initializing [ 4076.279330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4076.290748] Lustre: 121723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4076.419938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.725091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4082.740302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4082.859052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4083.754463] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4095.400787] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4101.173641] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4101.596548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4107.511451] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4107.866294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4113.895731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.908511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.929790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.947960] Lustre: Skipped 1 previous similar message [ 4116.543964] Lustre: server umount lustre-OST0000 complete [ 4121.570725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4121.585912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.613323] Lustre: Skipped 1 previous similar message [ 4121.626641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.248776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.258531] Lustre: Skipped 3 previous similar messages [ 4134.879128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4135.052240] Lustre: server umount lustre-MDT0000 complete [ 4138.468928] LustreError: 119767:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4138.495442] LustreError: 119767:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4138.590686] LustreError: 119747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130891 with bad export cookie 13206848960729393504 [ 4138.603031] LustreError: 119747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4138.609453] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.889825] Lustre: server umount lustre-MDT0001 complete [ 4149.063789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.819656] LustreError: 123943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.971630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.367031] LustreError: 123944:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.453746] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4159.459337] LustreError: 123943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.578219] LustreError: 123944:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.835858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.907794] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4182.492560] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.213828] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4196.682449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.292227] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4214.268156] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4226.633773] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4233.187832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4233.195060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.210808] Lustre: Skipped 1 previous similar message [ 4233.214991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.217775] Lustre: Skipped 2 previous similar messages [ 4237.154707] Lustre: server umount lustre-OST0000 complete [ 4243.938149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4243.946660] LustreError: Skipped 1 previous similar message [ 4243.949194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.969168] Lustre: Skipped 1 previous similar message [ 4247.825952] Lustre: server umount lustre-MDT0000 complete [ 4250.611613] LustreError: 123943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.649074] LustreError: 123943:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4253.035730] LustreError: 123924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131005 with bad export cookie 13206848960729394386 [ 4253.065746] LustreError: 123924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4253.084903] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4253.387904] Lustre: server umount lustre-MDT0001 complete [ 4268.254486] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4270.810372] Key type lgssc unregistered [ 4271.083216] LNet: 127578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4271.088941] LNetError: 127578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4272.102381] LNet: Removed LNI 192.168.202.136@tcp [ 4273.478329] Key type .llcrypt unregistered [ 4273.483110] Key type ._llcrypt unregistered [ 4294.663311] Key type ._llcrypt registered [ 4294.666226] Key type .llcrypt registered [ 4294.784720] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4312.185536] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4314.276690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4314.342240] alg: No test for adler32 (adler32-zlib) [ 4315.616759] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 4315.995594] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4317.703290] Key type lgssc registered [ 4319.510969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4361.428607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4361.473347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.847317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4362.892901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4363.015419] Lustre: lustre-MDT0000: new disk, initializing [ 4363.134095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.151198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4367.815642] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4382.650248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.952306] Lustre: 130739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4383.107742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4383.114161] Lustre: Skipped 1 previous similar message [ 4383.231813] Lustre: lustre-MDT0001: new disk, initializing [ 4383.487566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4383.607937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4383.635983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4390.117319] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4401.989954] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4408.918241] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4416.223750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4416.549861] Lustre: lustre-OST0000: new disk, initializing [ 4416.554239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4416.560807] Lustre: 132706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4416.637195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.758772] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4423.785977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4423.807570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4424.008128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4436.275254] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4441.741298] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4441.851634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4446.915360] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4447.191278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4449.764161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4449.771046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.774369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4449.780874] Lustre: Skipped 1 previous similar message [ 4449.795085] Lustre: Skipped 1 previous similar message [ 4454.883497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.891081] Lustre: Skipped 1 previous similar message [ 4460.011366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.026296] Lustre: Skipped 1 previous similar message [ 4463.010196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4463.020166] Lustre: Skipped 1 previous similar message [ 4463.071435] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4463.394793] Lustre: server umount lustre-OST0000 complete [ 4468.193641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.222178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.227723] Lustre: Skipped 1 previous similar message [ 4472.569376] Lustre: server umount lustre-MDT0000 complete [ 4477.184762] LustreError: 130729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131230 with bad export cookie 16650252357841954085 [ 4477.203512] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.208179] LustreError: 130729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4477.670552] Lustre: server umount lustre-MDT0001 complete [ 4486.531313] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:53:57 (1789131237) [ 4493.979570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.585818] LustreError: 135121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.609875] LustreError: 135121:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4494.659868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.941191] LustreError: 135122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.074548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4505.064418] LustreError: 135121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.201076] LustreError: 135122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.861674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.495382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4517.126171] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4525.747279] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4533.270522] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4536.740924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.743177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4536.761965] Lustre: Skipped 2 previous similar messages [ 4550.111131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4550.373914] Lustre: server umount lustre-MDT0000 complete [ 4552.164892] LustreError: 135118:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.188215] LustreError: 135118:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4553.997707] LustreError: 135103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131306 with bad export cookie 16650252357841954960 [ 4554.004503] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4554.263871] Lustre: server umount lustre-MDT0001 complete [ 4566.137507] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:55:15 (1789131315) [ 4576.577670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4592.351660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4597.471469] LustreError: 137538:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 4623.199842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.644216] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4640.595197] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4642.893233] Lustre: server umount lustre-OST0000 complete [ 4651.949824] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:56:42 (1789131402) [ 4659.257035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.872240] LustreError: 138985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.996131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.449730] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4673.082032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.921279] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4685.548915] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4692.497354] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4699.472395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4699.838993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.844047] Lustre: Skipped 1 previous similar message [ 4706.986197] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4716.661690] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4728.811799] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.836586] Lustre: Skipped 2 previous similar messages [ 4728.842697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4728.856080] Lustre: Skipped 7 previous similar messages [ 4731.574716] Lustre: server umount lustre-MDT0000 complete [ 4733.929469] LustreError: 140002:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.958637] LustreError: 140002:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4735.995817] LustreError: 138967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131488 with bad export cookie 16650252357841955632 [ 4736.011849] LustreError: 138967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4736.032494] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.694488] Lustre: server umount lustre-MDT0001 complete [ 4751.665898] Lustre: server umount lustre-OST0000 complete [ 4763.345912] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:58:33 (1789131513) [ 4772.831000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.563556] LustreError: 142641:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.641724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.641774] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4789.714068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.132694] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4804.685980] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.291578] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4819.445417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4827.970517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4836.224750] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4842.170209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4843.453308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4844.689184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4850.542829] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4850.819929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4853.735633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.746735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4853.747353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4853.747359] LustreError: Skipped 1 previous similar message [ 4853.753493] Lustre: Skipped 1 previous similar message [ 4853.778486] Lustre: Skipped 1 previous similar message [ 4859.333335] Lustre: server umount lustre-OST0000 complete [ 4863.968487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.989586] Lustre: Skipped 1 previous similar message [ 4869.831447] Lustre: server umount lustre-MDT0000 complete [ 4873.395943] LustreError: 143377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131626 with bad export cookie 16650252357841956703 [ 4873.402183] LustreError: 143377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4873.405056] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4873.680105] Lustre: server umount lustre-MDT0001 complete [ 4883.003782] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:00:33 (1789131633) [ 4892.252427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4907.935708] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4913.055764] LustreError: 146815:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 4938.784163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.798347] Lustre: Skipped 2 previous similar messages [ 4948.756640] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4957.844329] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4966.248227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.511894] LustreError: 146840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.524895] LustreError: 146840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4966.653674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4972.364334] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4983.663469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.054699] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5000.273974] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5008.405762] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5013.864504] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5014.100306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5019.728400] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5019.900751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5025.262430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.264544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5025.279573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5025.279583] Lustre: Skipped 6 previous similar messages [ 5025.288242] Lustre: Skipped 1 previous similar message [ 5028.753807] Lustre: server umount lustre-MDT0000 complete [ 5034.422515] LustreError: 146823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131787 with bad export cookie 16650252357841957725 [ 5034.430817] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5034.441608] LustreError: 146823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5035.037604] Lustre: server umount lustre-MDT0001 complete [ 5050.311309] Lustre: server umount lustre-OST0000 complete [ 5060.111551] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:03:30 (1789131810) [ 5067.957684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.552200] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5088.672261] LustreError: 151030:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 5123.499561] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5134.552618] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5142.213912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.682390] LustreError: 151055:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.708396] LustreError: 151055:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5142.814409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5148.067888] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5159.769348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.913712] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5176.496326] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5184.302538] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5192.682717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5193.063175] Lustre: lustre-OST0001: new disk, initializing [ 5193.067900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5193.079949] Lustre: 153902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5198.923410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5198.940727] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5199.133576] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5201.178374] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5213.395777] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5218.460395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5218.763194] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5223.288964] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5223.720848] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5226.980825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5227.002381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.017753] Lustre: Skipped 1 previous similar message [ 5227.023809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5227.026279] Lustre: Skipped 2 previous similar messages [ 5232.585985] Lustre: server umount lustre-OST0000 complete [ 5240.307135] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.326667] LustreError: Skipped 1 previous similar message [ 5245.683810] Lustre: server umount lustre-OST0001 complete [ 5252.576183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5252.583400] LustreError: Skipped 1 previous similar message [ 5257.784881] Lustre: server umount lustre-MDT0000 complete [ 5262.791384] LustreError: 155851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132015 with bad export cookie 16650252357841958677 [ 5262.796445] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5262.812493] LustreError: 155851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5263.497047] Lustre: server umount lustre-MDT0001 complete [ 5294.179304] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:07:24 (1789132044) [ 5296.862670] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5299.064944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5301.244597] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:07:31 (1789132051) [ 5309.785631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.045701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5310.104651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5310.292316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5310.302684] Lustre: Skipped 6 previous similar messages [ 5315.230285] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5326.036248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.273936] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5326.336497] Lustre: 157468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5333.719627] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5343.125512] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5350.614459] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5359.023079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5359.152912] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5366.200524] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5369.557717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5376.746698] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5383.580587] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5383.945408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5389.052787] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5389.305758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5395.425663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.435123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5395.446101] Lustre: Skipped 5 previous similar messages [ 5397.757986] Lustre: server umount lustre-OST0000 complete [ 5426.817184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5435.630692] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.670752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.671097] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5436.937316] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5446.481963] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5453.661850] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5453.955170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5460.118631] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5460.478781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5467.915599] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b3286db3000.ost_server_uuid 50 [ 5469.627098] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b3286db3000.ost_server_uuid in IDLE state after 0 sec [ 5469.825090] LustreError: 158602:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.36@tcp) returned error from blocking AST (req@00000000994f96bd x1876039899047936 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b5e1a9eb/0xe711944d79c7c06f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.36@tcp remote: 0x1b92df027b1eed2a expref: 5 pid: 157478 timeout: 5569 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5469.852131] LustreError: lustre-MDT0000: A client on nid 192.168.202.36@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5469.868504] LustreError: 157466:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.36@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b5e1a9eb/0xe711944d79c7c06f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.36@tcp remote: 0x1b92df027b1eed2a expref: 6 pid: 157478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5475.821714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5475.834697] LustreError: Skipped 1 previous similar message [ 5481.795717] Lustre: server umount lustre-OST0000 complete [ 5487.075574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.089521] Lustre: Skipped 13 previous similar messages [ 5493.409138] Lustre: server umount lustre-MDT0000 complete [ 5496.356792] LustreError: 157478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.371815] LustreError: 157478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [ 5497.762406] LustreError: 157461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132250 with bad export cookie 16650252357841959860 [ 5497.763740] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.797607] LustreError: 157461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5512.159595] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5512.792246] Lustre: server umount lustre-MDT0001 complete [ 5525.025218] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5528.239734] Key type lgssc unregistered [ 5528.644436] LNet: 163594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5528.650134] LNetError: 163594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5528.682737] LNet: Removed LNI 192.168.202.136@tcp [ 5529.655205] Key type .llcrypt unregistered [ 5529.658890] Key type ._llcrypt unregistered [ 5559.275935] Key type ._llcrypt registered [ 5559.277326] Key type .llcrypt registered [ 5559.464228] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5561.088570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5561.106844] alg: No test for adler32 (adler32-zlib) [ 5562.399221] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 5562.717983] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5564.407177] Key type lgssc registered [ 5565.758741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5574.742855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5574.773479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.235246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.063663] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5586.408439] LustreError: 165013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.438694] LustreError: 165013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5590.501954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.100826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.357715] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5605.297565] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5613.581738] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5622.582519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.923537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.438233] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5640.826951] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5646.454131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5652.081653] Lustre: Failing over lustre-MDT0000 [ 5652.341912] Lustre: server umount lustre-MDT0000 complete [ 5652.466427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.477741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.498775] LustreError: 165013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.056931] LustreError: 165741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.074735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.099191] LustreError: 165741:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5657.158485] Lustre: Skipped 1 previous similar message [ 5662.177937] LustreError: 165014:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.208590] LustreError: 165014:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5664.911678] LustreError: 164992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.36@tcp arrived at 1789132417 with bad export cookie 7409393133115133142 [ 5664.928743] LustreError: 164992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5667.301325] LustreError: 165013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.347141] LustreError: 165013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5673.442317] Lustre: 164325:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132410/real 1789132410] req@000000002aabda85 x1876041206365184/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789132426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5673.461884] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.239757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.560993] LustreError: 165741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.596927] LustreError: 165741:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5684.072605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.141038] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5689.085589] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5689.328113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5689.342656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5691.768447] Lustre: lustre-MDT0000: Denying connection for new client 5f7c4d65-3923-470d-94af-4b2659b6f64b (at 192.168.202.36@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5702.113819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5702.138323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.167260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5704.680827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5704.686935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.984805] Lustre: server umount lustre-OST0000 complete [ 5712.867541] LustreError: 168750:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5712.890278] Lustre: 168099:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5712.910421] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5712.995970] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5713.043239] LustreError: 168099:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000036ede932 x1876041206386688/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5713.095442] LustreError: 168099:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5713.151204] LustreError: 168099:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5713.172845] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5713.903905] Lustre: server umount lustre-MDT0000 complete [ 5714.922275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.938625] LustreError: 165008:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.977571] LustreError: 165008:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5720.251918] LustreError: 164993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132473 with bad export cookie 7409393133115133471 [ 5720.254556] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5720.263551] LustreError: 164993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5726.648568] Lustre: server umount lustre-MDT0001 complete [ 5739.063899] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5741.588934] Key type lgssc unregistered [ 5741.792492] LNet: 169506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5741.801364] LNetError: 169506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5741.813960] LNet: Removed LNI 192.168.202.136@tcp [ 5742.644943] Key type .llcrypt unregistered [ 5742.646714] Key type ._llcrypt unregistered [ 5774.163544] Key type ._llcrypt registered [ 5774.165207] Key type .llcrypt registered [ 5774.407872] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5776.458700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5776.521063] alg: No test for adler32 (adler32-zlib) [ 5778.069525] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 5778.633187] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5780.487183] Key type lgssc registered [ 5782.552426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5791.876429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5791.938753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.641280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.619730] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5804.001687] LustreError: 170921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.024390] LustreError: 170921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5808.434708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.808248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5814.643161] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5823.992741] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5831.042401] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5838.379120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.863709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.960471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5846.370823] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5853.998748] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5863.409119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.427975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.522821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.538538] Lustre: Skipped 2 previous similar messages [ 5869.026637] Lustre: server umount lustre-OST0000 complete [ 5873.632976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.638421] Lustre: Skipped 1 previous similar message [ 5873.641641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.646096] Lustre: Skipped 1 previous similar message [ 5878.753465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.768983] Lustre: Skipped 1 previous similar message [ 5879.895669] Lustre: server umount lustre-MDT0000 complete [ 5883.875025] LustreError: 171880:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.891929] LustreError: 171880:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5884.158104] LustreError: 171656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132637 with bad export cookie 17885273688329922334 [ 5884.168463] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.173548] LustreError: 171656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5884.930916] Lustre: server umount lustre-MDT0001 complete [ 5898.286965] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5902.560513] Key type lgssc unregistered [ 5903.238116] LNet: 174459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5903.253699] LNetError: 174459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5904.301695] LNet: Removed LNI 192.168.202.136@tcp [ 5905.426784] Key type .llcrypt unregistered [ 5905.429708] Key type ._llcrypt unregistered [ 5943.625889] Key type ._llcrypt registered [ 5943.629120] Key type .llcrypt registered [ 5943.838033] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5945.165422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5945.192039] alg: No test for adler32 (adler32-zlib) [ 5946.592237] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 5947.046996] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5948.800678] Key type lgssc registered [ 5950.337733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.206394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5964.280933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.067414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.572811] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5975.553956] LustreError: 176169:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.580073] LustreError: 176169:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5980.475453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.645980] LustreError: 176170:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.120801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5986.768202] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5995.550893] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6002.860512] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6011.817651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.286857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.484698] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6030.921498] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6035.828681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6046.182246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.182824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.194465] LustreError: Skipped 1 previous similar message [ 6046.204434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.214353] Lustre: Skipped 1 previous similar message [ 6049.896782] Lustre: server umount lustre-OST0000 complete [ 6056.423811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.427840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6056.435807] Lustre: Skipped 2 previous similar messages [ 6056.457238] Lustre: Skipped 1 previous similar message [ 6060.466781] Lustre: server umount lustre-MDT0000 complete [ 6061.540915] LustreError: 176899:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.558311] LustreError: 176899:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6064.575610] LustreError: 176149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132817 with bad export cookie 16484052917581848592 [ 6064.581983] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6064.587575] LustreError: 176149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6065.045030] Lustre: server umount lustre-MDT0001 complete [ 6078.248809] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6081.436869] Key type lgssc unregistered [ 6081.798945] LNet: 179720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6081.811389] LNetError: 179720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6081.837731] LNet: Removed LNI 192.168.202.136@tcp [ 6082.957319] Key type .llcrypt unregistered [ 6082.959593] Key type ._llcrypt unregistered [ 6112.103415] Key type ._llcrypt registered [ 6112.111443] Key type .llcrypt registered [ 6112.276594] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6113.783045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6113.842649] alg: No test for adler32 (adler32-zlib) [ 6114.907736] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6115.155642] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6116.847203] Key type lgssc registered [ 6118.649351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6133.583575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6133.637198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.055271] Lustre: *** cfs_fail_loc=135, val=0*** [ 6135.167359] LustreError: 181246:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6135.180734] LustreError: 181246:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -2 [ 6135.193133] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6135.193133] [ 6135.197745] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6135.215228] LustreError: 181206:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6135.226988] LustreError: 181206:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6135.233859] LustreError: 181206:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6135.243143] LustreError: 181206:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6135.347181] Lustre: server umount lustre-MDT0000 complete [ 6135.352311] LustreError: 181206:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6145.103251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.845133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.347845] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6155.237538] LustreError: 181618:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.264483] LustreError: 181618:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6160.355640] LustreError: 181619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.775535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.556366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6170.943050] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6180.641052] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6188.482470] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6195.169799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.180255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6195.182099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.188439] Lustre: Skipped 1 previous similar message [ 6197.903659] Lustre: server umount lustre-MDT0000 complete [ 6200.296964] LustreError: 181619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.316511] LustreError: 181619:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6202.426042] LustreError: 181600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132955 with bad export cookie 10848560354630035771 [ 6202.440793] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.941320] Lustre: server umount lustre-MDT0001 complete [ 6217.975555] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6222.225885] Key type lgssc unregistered [ 6222.676936] LNet: 183942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6222.691036] LNetError: 183942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6222.717466] LNet: Removed LNI 192.168.202.136@tcp [ 6224.037842] Key type .llcrypt unregistered [ 6224.041521] Key type ._llcrypt unregistered [ 6255.375276] Key type ._llcrypt registered [ 6255.389168] Key type .llcrypt registered [ 6255.571364] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6256.011729] Key type .llcrypt unregistered [ 6256.013410] Key type ._llcrypt unregistered [ 6276.560634] Key type ._llcrypt registered [ 6276.562721] Key type .llcrypt registered [ 6276.654275] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6278.464264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6278.603445] alg: No test for adler32 (adler32-zlib) [ 6279.906447] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6280.285117] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6282.047395] Key type lgssc registered [ 6283.751723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6293.180783] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6293.203572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.767674] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6313.887550] LustreError: 186081:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 6340.703248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.062517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6359.562799] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6369.417643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.799658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.880935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6374.881114] LustreError: 186106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.902971] LustreError: 186106:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6375.531956] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6375.916929] LustreError: 187119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.942755] LustreError: 187119:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6381.027689] LustreError: 187408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.038799] LustreError: 187408:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6386.146130] LustreError: 187116:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.183856] LustreError: 187116:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6386.661195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.480732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6392.736125] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6402.014598] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6410.025721] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6413.649338] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6426.084539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.097092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.133180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6428.641479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6428.642917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6431.546421] Lustre: server umount lustre-OST0000 complete [ 6438.881705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6438.882754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.907042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6438.912019] Lustre: Skipped 1 previous similar message [ 6443.805304] Lustre: server umount lustre-MDT0000 complete [ 6444.000916] LustreError: 187122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.025955] LustreError: 187122:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6449.906593] LustreError: 188747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133202 with bad export cookie 2951620844283794225 [ 6449.919880] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.934280] LustreError: 188747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6450.486588] Lustre: server umount lustre-MDT0001 complete [ 6466.367275] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6470.301699] Key type lgssc unregistered [ 6470.646588] LNet: 190057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6470.663532] LNetError: 190057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.694446] LNet: Removed LNI 192.168.202.136@tcp [ 6471.901639] Key type .llcrypt unregistered [ 6471.907059] Key type ._llcrypt unregistered [ 6499.463867] Key type ._llcrypt registered [ 6499.466664] Key type .llcrypt registered [ 6499.600177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6501.769231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.834654] alg: No test for adler32 (adler32-zlib) [ 6503.058585] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6503.512792] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6505.239621] Key type lgssc registered [ 6506.966095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6518.083974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6518.116513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.659128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6525.063792] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6530.039046] LustreError: 191471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.053173] LustreError: 191471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6535.139422] LustreError: 191472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.262224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.758302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.224523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6550.442119] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6557.749698] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6569.489808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6569.992803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.177650] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6589.462691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6592.812482] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6597.715826] Lustre: Failing over lustre-MDT0000 [ 6597.862642] Lustre: server umount lustre-MDT0000 complete [ 6599.659807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.671583] LustreError: 191466:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.681161] Lustre: Skipped 2 previous similar messages [ 6599.697740] LustreError: 191466:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6604.768637] LustreError: 191472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.793810] LustreError: 191472:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6609.889511] LustreError: 191471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.924788] LustreError: 191471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6616.033189] Lustre: 190782:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133352/real 1789133352] req@0000000042a67fb8 x1876042193072640/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789133368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6616.081472] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.825274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.218414] LustreError: 192570:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.228677] LustreError: 192570:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 6626.280562] LustreError: 190780:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000296c91fa x1876042193078016/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6626.952861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6627.005662] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6630.446825] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6632.276468] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6632.437762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6632.449432] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6632.512661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6638.749275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6644.517164] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6651.884909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.888520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.900628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.931017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.947533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.612478] Lustre: server umount lustre-OST0000 complete [ 6663.151862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6663.151987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6663.152585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6663.202758] Lustre: Skipped 2 previous similar messages [ 6665.064735] Lustre: server umount lustre-MDT0000 complete [ 6668.257326] LustreError: 192413:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.306966] LustreError: 192413:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 6670.864424] LustreError: 191453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133423 with bad export cookie 12287636433056254484 [ 6670.885582] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6670.892792] LustreError: 191453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6671.395571] Lustre: server umount lustre-MDT0001 complete [ 6685.749831] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6689.187275] Key type lgssc unregistered [ 6689.558643] LNet: 196271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6689.565862] LNetError: 196271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6689.577305] LNet: Removed LNI 192.168.202.136@tcp [ 6690.485768] Key type .llcrypt unregistered [ 6690.487446] Key type ._llcrypt unregistered [ 6717.398484] Key type ._llcrypt registered [ 6717.400827] Key type .llcrypt registered [ 6717.541043] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6719.464326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6719.535171] alg: No test for adler32 (adler32-zlib) [ 6720.708383] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6720.903363] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6722.561452] Key type lgssc registered [ 6724.110059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6734.028118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6734.082320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.752219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.429822] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6745.072873] LustreError: 197686:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.101086] LustreError: 197686:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6750.189376] LustreError: 197687:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.214890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.778680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6756.967422] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6764.496420] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6771.335846] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6779.124961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6779.680858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.763123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6787.259395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6795.799638] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6799.737296] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6806.724250] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6825.363958] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6837.219144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6837.223064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.226319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6837.236412] LustreError: Skipped 1 previous similar message [ 6837.245304] Lustre: Skipped 1 previous similar message [ 6839.897905] Lustre: server umount lustre-OST0000 complete [ 6843.881912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6843.887301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.918147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.927463] Lustre: Skipped 1 previous similar message [ 6847.456253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.460112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6847.479271] Lustre: Skipped 1 previous similar message [ 6850.018808] Lustre: server umount lustre-MDT0000 complete [ 6852.580632] LustreError: 198415:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.596487] LustreError: 198415:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6853.983434] LustreError: 200506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133606 with bad export cookie 11171991210921156562 [ 6853.986992] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6853.990356] LustreError: 200506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6854.454220] Lustre: server umount lustre-MDT0001 complete [ 6867.073867] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6870.005356] Key type lgssc unregistered [ 6870.337537] LNet: 201564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6870.353389] LNetError: 201564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6870.374247] LNet: Removed LNI 192.168.202.136@tcp [ 6871.063489] Key type .llcrypt unregistered [ 6871.065336] Key type ._llcrypt unregistered [ 6889.569405] Key type ._llcrypt registered [ 6889.572766] Key type .llcrypt registered [ 6889.767589] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6891.415493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6891.610607] alg: No test for adler32 (adler32-zlib) [ 6892.840379] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6893.357389] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6895.119412] Key type lgssc registered [ 6897.000991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6909.094170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6909.129558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.829112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.286894] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6920.163068] LustreError: 202832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.212720] LustreError: 202832:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6925.287585] LustreError: 202833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.300951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.076960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6933.252697] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6942.080276] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6949.307854] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6958.106786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.571237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6964.780923] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6974.794984] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6980.148223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6988.599523] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7000.328671] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7006.581530] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7017.043324] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7026.151864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7026.167089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7026.171327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.273220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.281821] Lustre: Skipped 2 previous similar messages [ 7032.601911] Lustre: server umount lustre-OST0000 complete [ 7040.514866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7040.530646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.552302] Lustre: Skipped 1 previous similar message [ 7040.565914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.143701] Lustre: server umount lustre-MDT0000 complete [ 7046.625305] LustreError: 202829:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.647785] LustreError: 202829:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7048.479701] LustreError: 204679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133801 with bad export cookie 1692250011745022530 [ 7048.481599] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.497165] LustreError: 204679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7048.849058] Lustre: server umount lustre-MDT0001 complete [ 7059.547465] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7062.160839] Key type lgssc unregistered [ 7062.418693] LNet: 207573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7062.424495] LNetError: 207573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7062.438187] LNet: Removed LNI 192.168.202.136@tcp [ 7063.423710] Key type .llcrypt unregistered [ 7063.427207] Key type ._llcrypt unregistered [ 7088.193896] Key type ._llcrypt registered [ 7088.198760] Key type .llcrypt registered [ 7088.347697] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7090.207962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7090.527154] alg: No test for adler32 (adler32-zlib) [ 7091.828317] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7092.224676] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7094.015238] Key type lgssc registered [ 7095.763927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.293386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7105.317253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.694598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.867792] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7116.280985] LustreError: 208993:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7116.317591] LustreError: 208993:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7121.376792] LustreError: 208994:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.287745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.875052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7131.241745] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7139.803985] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.393945] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.747563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.170759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.250865] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7172.178550] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7176.699672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7195.105848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.111950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.120177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7197.167559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7197.170496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7197.179555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.114173] Lustre: server umount lustre-OST0000 complete [ 7206.880755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7206.900705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.916869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.921807] Lustre: Skipped 1 previous similar message [ 7209.371424] Lustre: server umount lustre-MDT0000 complete [ 7212.515283] LustreError: 211439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.534404] LustreError: 211439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7213.009730] LustreError: 208976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133965 with bad export cookie 1260400529141545519 [ 7213.016067] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7213.020831] LustreError: 208976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7213.388289] Lustre: server umount lustre-MDT0001 complete [ 7225.349445] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7228.894460] Key type lgssc unregistered [ 7229.208742] LNet: 212683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7229.222264] LNetError: 212683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7229.253848] LNet: Removed LNI 192.168.202.136@tcp [ 7230.523593] Key type .llcrypt unregistered [ 7230.527032] Key type ._llcrypt unregistered [ 7254.702905] Key type ._llcrypt registered [ 7254.704534] Key type .llcrypt registered [ 7254.881934] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7256.214336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7256.260851] alg: No test for adler32 (adler32-zlib) [ 7257.354392] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7257.719429] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7259.495366] Key type lgssc registered [ 7260.902186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7269.482478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7269.498849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.910482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7275.521204] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7280.098774] LustreError: 213950:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.138805] LustreError: 213950:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7285.228770] LustreError: 213951:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7285.804616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7286.323091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7293.012274] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7303.223211] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7309.897892] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7316.466619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7317.041577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7323.467671] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7333.939426] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7340.630271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7355.877588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7355.881968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.919358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7355.929270] Lustre: Skipped 1 previous similar message [ 7360.629440] Lustre: server umount lustre-OST0000 complete [ 7366.114360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.119650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.124487] Lustre: Skipped 2 previous similar messages [ 7366.134353] Lustre: Skipped 1 previous similar message [ 7370.709636] Lustre: server umount lustre-MDT0000 complete [ 7371.260836] LustreError: 213947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.330623] LustreError: 213947:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7374.989850] LustreError: 213933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134127 with bad export cookie 9621985851113296512 [ 7375.006912] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7375.389264] Lustre: server umount lustre-MDT0001 complete [ 7386.451177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7389.098493] Key type lgssc unregistered [ 7389.335399] LNet: 217659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7389.340136] LNetError: 217659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7389.354386] LNet: Removed LNI 192.168.202.136@tcp [ 7389.978907] Key type .llcrypt unregistered [ 7389.982387] Key type ._llcrypt unregistered [ 7416.514582] Key type ._llcrypt registered [ 7416.527795] Key type .llcrypt registered [ 7416.822891] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7419.005642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7419.091824] alg: No test for adler32 (adler32-zlib) [ 7420.215429] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7420.509796] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7422.168740] Key type lgssc registered [ 7423.871387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7432.406191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7432.428319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.894781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.098764] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7443.425970] LustreError: 218927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.448639] LustreError: 218927:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7447.800367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.168285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7453.906446] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7461.999918] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7468.231968] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7477.115712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.446815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.452743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7483.856804] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7492.497732] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7501.171389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7501.234922] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7501.471001] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.771788] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7515.254484] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7517.381151] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7529.765118] Lustre: Permanently deactivating lustre-OST0001 [ 7529.791458] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7535.643448] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7541.734523] Lustre: Permanently reactivating lustre-OST0001 [ 7541.751093] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7541.764776] Lustre: Skipped 2 previous similar messages [ 7543.261767] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.294953] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7543.302525] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7543.329796] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7543.420080] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7557.616931] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.639932] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7557.642831] Lustre: Skipped 2 previous similar messages [ 7562.732029] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7562.743426] Lustre: Skipped 2 previous similar messages [ 7567.862454] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7567.870881] Lustre: Skipped 1 previous similar message [ 7568.351185] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7568.694820] Lustre: server umount lustre-OST0001 complete [ 7572.966204] LustreError: 222485:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.002377] LustreError: 222485:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7576.036186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7576.037216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.038773] LustreError: Skipped 1 previous similar message [ 7576.048273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7576.061711] Lustre: Skipped 1 previous similar message [ 7578.088871] LustreError: 220750:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.108675] LustreError: 220750:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7579.599935] Lustre: server umount lustre-OST0000 complete [ 7586.272258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7586.276221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.320336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7586.331782] Lustre: Skipped 3 previous similar messages [ 7590.783849] Lustre: server umount lustre-MDT0000 complete [ 7593.441871] LustreError: 220811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.477328] LustreError: 220811:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7596.762829] LustreError: 221766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134349 with bad export cookie 2930617271490960851 [ 7596.776083] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7596.796078] LustreError: 221766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7597.494811] Lustre: server umount lustre-MDT0001 complete [ 7612.686527] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7615.487327] Key type lgssc unregistered [ 7615.830514] LNet: 223973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7615.835896] LNetError: 223973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7615.853065] LNet: Removed LNI 192.168.202.136@tcp [ 7616.808744] Key type .llcrypt unregistered [ 7616.815581] Key type ._llcrypt unregistered [ 7659.552233] Key type ._llcrypt registered [ 7659.554978] Key type .llcrypt registered [ 7659.684245] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7661.114116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7661.136637] alg: No test for adler32 (adler32-zlib) [ 7662.144968] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7662.361905] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7664.007235] Key type lgssc registered [ 7665.241798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7673.683856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7673.704602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.050448] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7675.062979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7675.257555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.912661] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7691.632098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7691.690475] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7691.742335] Lustre: 226181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7691.991438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7698.127920] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7706.960287] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7715.809838] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7723.965456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.286332] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7724.499832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7731.426845] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7734.071450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7741.634958] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7746.627977] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7749.596535] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7759.344766] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7767.398503] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7782.658680] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7782.664932] Lustre: Skipped 1 previous similar message [ 7805.327573] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7805.331706] Lustre: Skipped 2 previous similar messages [ 7845.818942] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7845.824491] Lustre: Skipped 4 previous similar messages [ 7913.444694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.452086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7913.489311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7913.499951] Lustre: Skipped 1 previous similar message [ 7917.046494] Lustre: server umount lustre-OST0000 complete [ 7922.660593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7922.671954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.688538] Lustre: Skipped 1 previous similar message [ 7922.699716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7923.708789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.741713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7923.757945] Lustre: Skipped 1 previous similar message [ 7928.705689] Lustre: server umount lustre-MDT0000 complete [ 7933.652675] LustreError: 226922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134686 with bad export cookie 8763639551318845761 [ 7933.655794] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7933.663645] LustreError: 226922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7934.073161] Lustre: server umount lustre-MDT0001 complete [ 7946.395983] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7949.468571] Key type lgssc unregistered [ 7949.784181] LNet: 230985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7949.791327] LNetError: 230985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7949.806284] LNet: Removed LNI 192.168.202.136@tcp [ 7950.746129] Key type .llcrypt unregistered [ 7950.747843] Key type ._llcrypt unregistered [ 7976.362161] Key type ._llcrypt registered [ 7976.364413] Key type .llcrypt registered [ 7976.488514] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7977.452241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7977.506357] alg: No test for adler32 (adler32-zlib) [ 7978.619792] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7978.925586] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7980.671201] Key type lgssc registered [ 7981.405736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7991.158983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7991.193408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.078891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7997.741070] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8002.554926] LustreError: 232403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.581140] LustreError: 232403:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8007.197880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.623201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8011.681799] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8021.400048] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8029.620411] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.988390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.796062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.469669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8062.225683] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8062.283478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 8077.795203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.799603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.819536] Lustre: Skipped 1 previous similar message [ 8080.159107] Lustre: server umount lustre-OST0000 complete [ 8088.051722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.070534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8088.073533] Lustre: Skipped 1 previous similar message [ 8092.197595] Lustre: server umount lustre-MDT0000 complete [ 8093.152365] LustreError: 232402:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.180785] LustreError: 232402:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8096.954387] LustreError: 232382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134849 with bad export cookie 1851681600169975540 [ 8096.966341] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8096.972322] LustreError: 232382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8097.266192] Lustre: server umount lustre-MDT0001 complete [ 8111.035315] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8113.923497] Key type lgssc unregistered [ 8114.157983] LNet: 236098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8114.178885] LNetError: 236098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8114.209905] LNet: Removed LNI 192.168.202.136@tcp [ 8115.292730] Key type .llcrypt unregistered [ 8115.295061] Key type ._llcrypt unregistered [ 8144.861258] Key type ._llcrypt registered [ 8144.865714] Key type .llcrypt registered [ 8144.976860] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8146.258482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8146.267580] alg: No test for adler32 (adler32-zlib) [ 8147.325453] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8147.536638] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8149.191179] Key type lgssc registered [ 8150.862296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8424.539199] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8427.777561] Key type lgssc unregistered [ 8428.256336] LNet: 238125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8428.259260] LNetError: 238125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8428.296919] LNet: Removed LNI 192.168.202.136@tcp [ 8429.130552] Key type .llcrypt unregistered [ 8429.132587] Key type ._llcrypt unregistered [ 8463.978339] Key type ._llcrypt registered [ 8463.983517] Key type .llcrypt registered [ 8464.102134] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8483.259710] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8484.486436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8484.509766] alg: No test for adler32 (adler32-zlib) [ 8485.818190] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8486.320735] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8488.088782] Key type lgssc registered [ 8489.713914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8527.511765] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8272 sec ========== 10:01:17 (1789135277) [ 8529.368838] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:01:19 (1789135279) === [ 8534.618747] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:01:24 (1789135284) === [ 8592.190335] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8595.327675] Key type lgssc unregistered [ 8595.605799] LNet: 243381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8595.610867] LNetError: 243381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8595.627295] LNet: Removed LNI 192.168.202.136@tcp [ 8596.559939] Key type .llcrypt unregistered [ 8596.563873] Key type ._llcrypt unregistered