[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454143924 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002276] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006673] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009141] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012077] *** VALIDATE selinux *** [ 0.019759] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023592] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026130] *** VALIDATE tmpfs *** [ 0.027473] *** VALIDATE proc *** [ 0.029201] *** VALIDATE cgroup *** [ 0.030010] *** VALIDATE cgroup2 *** [ 0.031271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033112] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035028] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039343] debug: unmapping init [mem 0xffffffff87659000-0xffffffff87660fff] [ 0.041186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042614] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045016] ... generic registers: 4 [ 0.046021] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048012] ... fixed-purpose events: 3 [ 0.049008] ... event mask: 000000070000000f [ 0.050284] rcu: Hierarchical SRCU implementation. [ 0.052384] smp: Bringing up secondary CPUs ... [ 0.053608] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057357] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134282] node 0 deferred pages initialised in 71ms [ 0.138098] devtmpfs: initialized [ 0.139325] x86/mm: Memory block size: 128MB [ 0.141785] gcov: version magic: 0x41383552 [ 0.144310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147317] pinctrl core: initialized pinctrl subsystem [ 0.149180] [ 0.149819] ************************************************************* [ 0.153013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155012] ** ** [ 0.157011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160012] ** ** [ 0.163012] ** This means that this kernel is built to expose internal ** [ 0.165012] ** IOMMU data structures, which may compromise security on ** [ 0.168011] ** your system. ** [ 0.170011] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.175013] ** kernel, report this immediately to your vendor! ** [ 0.178012] ** ** [ 0.180010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183012] ************************************************************* [ 0.186642] NET: Registered protocol family 16 [ 0.188410] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197089] cpuidle: using governor menu [ 0.198588] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201086] PCI: Using configuration type 1 for base access [ 0.203143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215048] cryptd: max_cpu_qlen set to 1000 [ 0.217220] ACPI: Added _OSI(Module Device) [ 0.219013] ACPI: Added _OSI(Processor Device) [ 0.221018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228517] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234525] ACPI: Interpreter enabled [ 0.236060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237012] ACPI: Using IOAPIC for interrupt routing [ 0.239182] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243742] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254450] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267000] acpiphp: Slot [2] registered [ 0.267000] acpiphp: Slot [5] registered [ 0.269142] acpiphp: Slot [6] registered [ 0.271085] acpiphp: Slot [7] registered [ 0.272102] acpiphp: Slot [8] registered [ 0.274129] acpiphp: Slot [9] registered [ 0.275107] acpiphp: Slot [10] registered [ 0.277103] acpiphp: Slot [3] registered [ 0.279111] acpiphp: Slot [4] registered [ 0.280097] acpiphp: Slot [11] registered [ 0.282092] acpiphp: Slot [12] registered [ 0.283084] acpiphp: Slot [13] registered [ 0.285061] acpiphp: Slot [14] registered [ 0.286011] acpiphp: Slot [15] registered [ 0.286988] acpiphp: Slot [16] registered [ 0.287065] acpiphp: Slot [17] registered [ 0.288078] acpiphp: Slot [18] registered [ 0.290325] acpiphp: Slot [19] registered [ 0.291094] acpiphp: Slot [20] registered [ 0.293090] acpiphp: Slot [21] registered [ 0.295094] acpiphp: Slot [22] registered [ 0.296103] acpiphp: Slot [23] registered [ 0.298088] acpiphp: Slot [24] registered [ 0.299085] acpiphp: Slot [25] registered [ 0.301086] acpiphp: Slot [26] registered [ 0.303087] acpiphp: Slot [27] registered [ 0.304088] acpiphp: Slot [28] registered [ 0.306101] acpiphp: Slot [29] registered [ 0.307063] acpiphp: Slot [30] registered [ 0.309053] acpiphp: Slot [31] registered [ 0.310051] PCI host bridge to bus 0000:00 [ 0.312017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.327197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345812] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.355013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358097] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361390] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366236] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393008] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.399015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405905] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.513460] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.526014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.559000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.570015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.578015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.658525] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.666014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.683013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.704016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.716195] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718489] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725969] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734081] iommu: Default domain type: Passthrough [ 0.736529] SCSI subsystem initialized [ 0.737074] ACPI: bus type USB registered [ 0.738110] usbcore: registered new interface driver usbfs [ 0.740054] usbcore: registered new interface driver hub [ 0.742055] usbcore: registered new device driver usb [ 0.743132] pps_core: LinuxPPS API ver. 1 registered [ 0.746010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.749055] PTP clock support registered [ 0.752080] EDAC MC: Ver: 3.0.0 [ 0.754110] PCI: Using ACPI for IRQ routing [ 0.755856] NetLabel: Initializing [ 0.757010] NetLabel: domain hash size = 128 [ 0.759009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761072] NetLabel: unlabeled traffic allowed by default [ 0.763095] vgaarb: loaded [ 0.764250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.774000] clocksource: Switched to clocksource kvm-clock [ 0.873758] VFS: Disk quotas dquot_6.6.0 [ 0.875585] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.878668] *** VALIDATE ramfs *** [ 0.880461] *** VALIDATE hugetlbfs *** [ 0.883229] pnp: PnP ACPI init [ 0.885973] pnp: PnP ACPI: found 6 devices [ 0.907826] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.911478] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914010] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.916499] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919124] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921743] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924636] NET: Registered protocol family 2 [ 0.927075] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931811] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.935704] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.942183] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.945388] TCP: Hash tables configured (established 65536 bind 65536) [ 0.948102] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.951136] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953457] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956637] NET: Registered protocol family 1 [ 0.959848] RPC: Registered named UNIX socket transport module. [ 0.962465] RPC: Registered udp transport module. [ 0.964372] RPC: Registered tcp transport module. [ 0.965763] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.967980] NET: Registered protocol family 44 [ 0.969755] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.972200] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.974843] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978600] PCI: CLS 0 bytes, default 64 [ 0.980872] Unpacking initramfs... [ 2.349748] debug: unmapping init [mem 0xffff9642fcc54000-0xffff9642fffbffff] [ 2.353357] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.355424] software IO TLB: mapped [mem 0x00000000a3200000-0x00000000a7200000] (64MB) [ 2.357941] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.955841] Initialise system trusted keyrings [ 3.962228] Key type blacklist registered [ 3.965694] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.986163] zbud: loaded [ 3.992002] *** VALIDATE nfs *** [ 3.995650] *** VALIDATE nfs4 *** [ 4.002035] pstore: using deflate compression [ 4.041540] Platform Keyring initialized [ 4.280149] NET: Registered protocol family 38 [ 4.282687] Key type asymmetric registered [ 4.284756] Asymmetric key parser 'x509' registered [ 4.287516] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.291791] io scheduler mq-deadline registered [ 4.293858] io scheduler kyber registered [ 4.296417] io scheduler bfq registered [ 4.298832] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.303195] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.307649] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.313787] ACPI: Power Button [PWRF] [ 4.324707] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.378961] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.422309] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.441626] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.480230] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.524851] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.570676] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.599583] Non-volatile memory driver v1.3 [ 4.612780] Linux agpgart interface v0.103 [ 4.733976] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 4.739979] vda: detected capacity change from 0 to 71311360 [ 4.782491] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.790414] vdb: detected capacity change from 0 to 1073741824 [ 4.838448] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.850806] vdc: detected capacity change from 0 to 2621440000 [ 4.907802] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.917324] vdd: detected capacity change from 0 to 2621440000 [ 4.969750] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.980871] vde: detected capacity change from 0 to 4294967296 [ 5.031887] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.036792] vdf: detected capacity change from 0 to 4294967296 [ 5.047407] libphy: Fixed MDIO Bus: probed [ 5.056648] usbcore: registered new interface driver usbserial_generic [ 5.060454] usbserial: USB Serial support registered for generic [ 5.063928] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.070614] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.073359] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.077829] mousedev: PS/2 mouse device common for all mice [ 5.084878] rtc_cmos 00:05: RTC can wake from S4 [ 5.090777] rtc_cmos 00:05: registered as rtc0 [ 5.093987] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.098099] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.100483] intel_pstate: CPU model not supported [ 5.105871] hid: raw HID events driver (C) Jiri Kosina [ 5.113904] usbcore: registered new interface driver usbhid [ 5.118986] usbhid: USB HID core driver [ 5.131156] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.135967] drop_monitor: Initializing network drop monitor service [ 5.145360] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.154573] Initializing XFRM netlink socket [ 5.164273] NET: Registered protocol family 10 [ 5.173360] Segment Routing with IPv6 [ 5.178632] NET: Registered protocol family 17 [ 5.189065] mpls_gso: MPLS GSO support [ 5.212761] RAS: Correctable Errors collector initialized. [ 5.231553] AVX version of gcm_enc/dec engaged. [ 5.241889] AES CTR mode by8 optimization enabled [ 5.499716] sched_clock: Marking stable (5499691572, 0)->(6346421740, -846730168) [ 5.508142] registered taskstats version 1 [ 5.531453] Loading compiled-in X.509 certificates [ 5.535593] zswap: loaded using pool lzo/zbud [ 5.601679] Key type big_key registered [ 5.627504] Key type encrypted registered [ 5.630585] ima: No TPM chip found, activating TPM-bypass! [ 5.633940] ima: Allocated hash algorithm: sha1 [ 5.637176] ima: No architecture policies found [ 5.640217] evm: Initialising EVM extended attributes: [ 5.642953] evm: security.selinux [ 5.645419] evm: security.ima [ 5.647201] evm: security.capability [ 5.649350] evm: HMAC attrs: 0x1 [ 5.653539] rtc_cmos 00:05: setting system clock to 2026-06-16 10:19:05 UTC (1781605145) [ 5.664607] debug: unmapping init [mem 0xffffffff88603000-0xffffffff887fffff] [ 5.669462] debug: unmapping init [mem 0xffffffff87382000-0xffffffff87658fff] [ 5.680080] Write protecting the kernel read-only data: 28672k [ 5.687513] debug: unmapping init [mem 0xffffffff85a03000-0xffffffff85bfffff] [ 5.693405] debug: unmapping init [mem 0xffffffff86314000-0xffffffff863fffff] [ 5.851641] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.882937] systemd[1]: Detected virtualization kvm. [ 5.892993] systemd[1]: Detected architecture x86-64. [ 5.895242] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.970975] systemd[1]: No hostname configured. [ 5.978853] systemd[1]: Set hostname to . [ 5.985950] random: systemd: uninitialized urandom read (16 bytes read) [ 5.990856] systemd[1]: Initializing machine ID from random generator. [ 6.120281] random: ln: uninitialized urandom read (6 bytes read) [ 6.504964] random: systemd: uninitialized urandom read (16 bytes read) [ 6.509956] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.524190] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 6.550970] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.763949] device-mapper: uevent: version 1.0.3 [ 8.769218] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.429758] virtio_net virtio0 ens2: renamed from eth0 [ 11.464710] random: fast init done [ 11.826677] scsi host0: ata_piix [ 12.044283] scsi host1: ata_piix [ 12.053369] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.079607] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.167268] random: crng init done [ 17.172622] random: 7 urandom warning(s) missed due to ratelimiting [ 20.450997] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.567762] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.146217] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.935105] SELinux: Disabled at runtime. [ 29.193757] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.226775] systemd[1]: Detected virtualization kvm. [ 29.230959] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.087714] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.093806] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.110784] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.137407] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.146352] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.165191] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.186658] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ 31.693958] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 32.923653] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.174533] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.210104] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.591556] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.626255] EDAC sbridge: Ver: 1.1.2 [ 38.565063] hrtimer: interrupt took 3148273 ns [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 39.678616] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 40.627601] NFS: Registering the id_resolver key type [ 40.632342] Key type id_resolver registered [ 40.634735] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg638-server login: [ 116.196857] libcfs: loading out-of-tree module taints kernel. [ 116.235204] Key type ._llcrypt registered [ 116.237433] Key type .llcrypt registered [ 116.350404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 140.521569] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 142.641428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 142.665825] alg: No test for adler32 (adler32-zlib) [ 144.367560] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 145.450251] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 147.416518] Key type lgssc registered [ 149.602732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 177.551774] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 232.622540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 252.156961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 252.192805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 253.688049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 253.739802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 253.957620] Lustre: lustre-MDT0000: new disk, initializing [ 254.097311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.191182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 261.228208] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.226626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 281.397817] Lustre: 6528:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 281.498800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 281.508752] Lustre: Skipped 1 previous similar message [ 281.663382] Lustre: lustre-MDT0001: new disk, initializing [ 281.859311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 281.934355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 281.970522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 287.975850] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.395308] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 305.887317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.190952] Lustre: lustre-OST0000: new disk, initializing [ 306.195346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 306.201950] Lustre: 8468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 306.287760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.898255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 312.911449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 313.028142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 313.973631] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 331.769370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 331.939897] Lustre: lustre-OST0001: new disk, initializing [ 331.947318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 331.955128] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 332.067031] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 339.054707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 339.076065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 339.171563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 340.986034] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 355.767169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 365.312173] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 373.713901] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing check_logdir /tmp/testlogs/ [ 380.193995] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing yml_node [ 386.028088] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 389.790082] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 393.016859] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 395.182204] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 06:25:34 EDT 2026 [ 417.943835] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 420.464471] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 440.800909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 440.826468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.852579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 441.825750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.827437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 441.853486] Lustre: Skipped 1 previous similar message [ 441.884331] Lustre: Skipped 3 previous similar messages [ 444.967297] Lustre: server umount lustre-MDT0000 complete [ 450.216321] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605590 with bad export cookie 14737626057845046581 [ 450.227648] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.229513] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 452.068333] LustreError: 6534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.069899] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.106655] LustreError: 6534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 452.129303] Lustre: Skipped 1 previous similar message [ 452.145170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 452.152886] Lustre: Skipped 1 previous similar message [ 455.072344] LustreError: 8043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.073981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 455.144412] Lustre: Skipped 1 previous similar message [ 456.694317] Lustre: server umount lustre-MDT0001 complete [ 462.016078] Lustre: server umount lustre-OST0000 complete [ 467.197705] Lustre: server umount lustre-OST0001 complete [ 475.141356] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 487.547262] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 536.329320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.984583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 537.053301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 537.159347] Lustre: lustre-MDT0000: new disk, initializing [ 537.363137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.406860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 544.253900] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 559.553995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.718347] Lustre: 15101:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 559.770259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 559.773940] Lustre: Skipped 1 previous similar message [ 559.962903] Lustre: lustre-MDT0001: new disk, initializing [ 560.200140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 560.237574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 560.248269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 566.306726] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 578.008610] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.800255] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 594.016593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.328722] Lustre: lustre-OST0000: new disk, initializing [ 594.333144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 594.339955] Lustre: 17071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 594.429967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.435032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 596.445343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 596.632328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 603.353716] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 615.670333] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 621.867358] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 622.213380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 629.554911] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 630.019529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 632.804402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 632.817145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.840727] Lustre: Skipped 1 previous similar message [ 632.853089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 637.409709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.419535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.185679] Lustre: server umount lustre-OST0000 complete [ 647.137464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 647.146714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.183231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.194729] Lustre: Skipped 2 previous similar messages [ 650.831583] Lustre: server umount lustre-MDT0000 complete [ 652.769910] LustreError: 15107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.798448] LustreError: 15107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 656.504296] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605796 with bad export cookie 14737626057845048114 [ 656.507157] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.519590] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 657.228331] Lustre: server umount lustre-MDT0001 complete [ 662.494866] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:30:01 (1781605801) [ 671.397906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.941409] LustreError: 19339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.023181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.348374] LustreError: 19340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.377093] LustreError: 19340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 677.971682] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 686.564278] LustreError: 19340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.595995] LustreError: 19340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 690.254452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.969218] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 708.952958] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 716.397364] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 725.473666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.078481] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 748.768118] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 761.313211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 761.337828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.370650] Lustre: Skipped 1 previous similar message [ 761.409740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 761.426725] Lustre: Skipped 1 previous similar message [ 764.615729] Lustre: server umount lustre-OST0000 complete [ 777.707301] LustreError: 19339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.034151] Lustre: server umount lustre-MDT0000 complete [ 785.931127] LustreError: 20076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605925 with bad export cookie 14737626057845048996 [ 785.948857] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.953091] LustreError: 20076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.429806] Lustre: server umount lustre-MDT0001 complete [ 806.380580] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 810.741047] Key type lgssc unregistered [ 811.360688] LNet: 22900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.369479] LNetError: 22900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.394083] LNet: Removed LNI 192.168.206.138@tcp [ 813.077495] Key type .llcrypt unregistered [ 813.083882] Key type ._llcrypt unregistered [ 845.561591] Key type ._llcrypt registered [ 845.570403] Key type .llcrypt registered [ 845.814857] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 847.222078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.294657] alg: No test for adler32 (adler32-zlib) [ 849.226849] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 849.738840] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 851.472602] Key type lgssc registered [ 853.645464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 865.268232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 865.319651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.174185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.044504] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 876.519277] LustreError: 24315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.552521] LustreError: 24315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 881.636955] LustreError: 24316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.766295] LustreError: 24315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.721741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.328159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 896.180347] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 910.184354] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 918.610843] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 927.670556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.255547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.447495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 936.538303] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 946.070983] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 966.479238] Lustre: server umount lustre-OST0000 complete [ 972.265656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.271309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.304533] Lustre: Skipped 1 previous similar message [ 972.323949] Lustre: Skipped 1 previous similar message [ 977.378218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.393072] Lustre: Skipped 1 previous similar message [ 980.550409] Lustre: server umount lustre-MDT0000 complete [ 982.507244] Lustre: 23626:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606106/real 1781606106] req@0000000035a7d22d x1868149281557888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781606122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 982.517699] LustreError: 24311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.542620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.568218] LustreError: 24311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 984.969672] LustreError: 24297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606124 with bad export cookie 8650359007178379866 [ 984.980723] LustreError: 24297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 984.980771] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.527585] Lustre: server umount lustre-MDT0001 complete [ 998.755466] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1002.478429] Key type lgssc unregistered [ 1002.880808] LNet: 28298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.896157] LNetError: 28298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.919211] LNet: Removed LNI 192.168.206.138@tcp [ 1004.058600] Key type .llcrypt unregistered [ 1004.060945] Key type ._llcrypt unregistered [ 1031.789178] Key type ._llcrypt registered [ 1031.799395] Key type .llcrypt registered [ 1031.927338] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1033.336583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.404238] alg: No test for adler32 (adler32-zlib) [ 1034.586651] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1034.821929] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1036.543713] Key type lgssc registered [ 1038.135281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1047.482403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1047.508718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.203157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.951520] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1058.802720] LustreError: 29714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.820064] LustreError: 29714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1063.909269] LustreError: 29715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.754686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.301049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.048332] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1077.374612] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1084.116282] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1097.675046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1098.071191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.509216] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1113.626235] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1118.587952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1125.857763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.872125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.886257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.930902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.941628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.950779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.985339] Lustre: Skipped 1 previous similar message [ 1130.296613] Lustre: server umount lustre-OST0000 complete [ 1137.122140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.130283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.149887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.172724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.174770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.223484] Lustre: Skipped 1 previous similar message [ 1140.661871] Lustre: server umount lustre-MDT0000 complete [ 1144.293109] LustreError: 29715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.306520] LustreError: 29715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1145.137919] LustreError: 29696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606284 with bad export cookie 4820407477802251579 [ 1145.158096] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.593523] Lustre: server umount lustre-MDT0001 complete [ 1156.992099] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1160.065589] Key type lgssc unregistered [ 1160.430656] LNet: 33593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.440309] LNetError: 33593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.461676] LNet: Removed LNI 192.168.206.138@tcp [ 1161.427792] Key type .llcrypt unregistered [ 1161.432706] Key type ._llcrypt unregistered [ 1184.968449] Key type ._llcrypt registered [ 1184.974880] Key type .llcrypt registered [ 1185.122884] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1186.566800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.744601] alg: No test for adler32 (adler32-zlib) [ 1187.888258] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1188.133634] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1189.871220] Key type lgssc registered [ 1191.154699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1199.572683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1199.590687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.235674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.133288] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1210.354328] LustreError: 35006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.371072] LustreError: 35006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1215.457276] LustreError: 35005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1216.062845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.491091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1221.288618] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1228.793692] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1234.399424] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1241.700040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.100653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.257690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1248.354450] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1257.146663] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1270.755434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1270.763561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.773147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1270.795512] Lustre: Skipped 1 previous similar message [ 1274.097500] Lustre: server umount lustre-OST0000 complete [ 1280.996429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.013859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.015278] Lustre: Skipped 1 previous similar message [ 1281.036958] Lustre: Skipped 2 previous similar messages [ 1284.419520] Lustre: server umount lustre-MDT0000 complete [ 1286.112502] LustreError: 35006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.133129] LustreError: 35006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1288.313286] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606428 with bad export cookie 14116131401573997679 [ 1288.321143] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1288.322156] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1288.568834] Lustre: server umount lustre-MDT0001 complete [ 1299.594504] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1302.579903] Key type lgssc unregistered [ 1302.901141] LNet: 38539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.908858] LNetError: 38539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.977852] LNet: Removed LNI 192.168.206.138@tcp [ 1304.807886] Key type .llcrypt unregistered [ 1304.810276] Key type ._llcrypt unregistered [ 1326.582727] Key type ._llcrypt registered [ 1326.585144] Key type .llcrypt registered [ 1326.704474] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1327.721052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.731507] alg: No test for adler32 (adler32-zlib) [ 1328.767760] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1328.959705] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1330.600062] Key type lgssc registered [ 1331.952845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1339.525806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1339.547766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.031471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1345.869472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1351.139870] LustreError: 39952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.168654] LustreError: 39952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1353.420423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.815150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.823737] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1365.040588] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1370.268468] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1376.701591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.984918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.606245] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1385.203493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1389.216972] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1395.684846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.692814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1395.710185] Lustre: Skipped 1 previous similar message [ 1397.162988] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1397.170077] Lustre: Skipped 1 previous similar message [ 1400.802392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.724346] Lustre: server umount lustre-OST0000 complete [ 1419.967354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.970399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.979155] Lustre: Skipped 1 previous similar message [ 1419.995653] Lustre: Skipped 1 previous similar message [ 1425.378256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.384450] Lustre: Skipped 2 previous similar messages [ 1426.261920] Lustre: server umount lustre-MDT0000 complete [ 1426.400802] LustreError: 39953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.430133] LustreError: 39953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1429.529926] LustreError: 39933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606569 with bad export cookie 3776221141461686623 [ 1429.535082] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1429.549524] LustreError: 39933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1435.947957] Lustre: server umount lustre-MDT0001 complete [ 1445.976465] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1448.823993] Key type lgssc unregistered [ 1449.111695] LNet: 43660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.126338] LNetError: 43660:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.162101] LNet: Removed LNI 192.168.206.138@tcp [ 1450.017875] Key type .llcrypt unregistered [ 1450.019345] Key type ._llcrypt unregistered [ 1470.200176] Key type ._llcrypt registered [ 1470.201956] Key type .llcrypt registered [ 1470.314214] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1471.624070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.668499] alg: No test for adler32 (adler32-zlib) [ 1472.736742] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1472.972116] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1474.631712] Key type lgssc registered [ 1475.819534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1483.239954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1483.275769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.705142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.985184] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1495.012052] LustreError: 45065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.021505] LustreError: 45065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1497.045371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.524133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.834570] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1508.657606] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1513.995773] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1520.323419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1520.763765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.193342] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1529.968873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1532.173344] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1538.532899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1538.541107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.550518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.482177] Lustre: server umount lustre-MDT0000 complete [ 1540.587925] LustreError: 45066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.594597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.164109] LustreError: 47643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606683 with bad export cookie 6344355039644870088 [ 1543.164109] LustreError: 45044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606683 with bad export cookie 6344355039644870088 [ 1543.165314] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.181530] LustreError: 45044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1544.094586] LustreError: 45061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.116406] LustreError: 45061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1544.128982] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 1544.140273] Lustre: Skipped 1 previous similar message [ 1545.697602] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.710399] Lustre: Skipped 1 previous similar message [ 1545.717630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1548.704377] LustreError: 45062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.704406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1548.723711] LustreError: 45062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1549.603494] Lustre: server umount lustre-MDT0001 complete [ 1569.926846] Lustre: server umount lustre-OST0000 complete [ 1584.834477] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1587.924677] Key type lgssc unregistered [ 1588.197772] LNet: 48925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.214083] LNetError: 48925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.240489] LNet: Removed LNI 192.168.206.138@tcp [ 1589.171923] Key type .llcrypt unregistered [ 1589.174769] Key type ._llcrypt unregistered [ 1610.201343] Key type ._llcrypt registered [ 1610.206773] Key type .llcrypt registered [ 1610.316318] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1611.393678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1611.442763] alg: No test for adler32 (adler32-zlib) [ 1612.504873] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1612.817881] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1614.495190] Key type lgssc registered [ 1615.578722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.398590] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1622.415258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.920401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1643.039307] LustreError: 50292:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 1669.727237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.460786] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1682.023649] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1846.590219] Lustre: server umount lustre-OST0000 complete [ 1860.650472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1862.940803] Key type lgssc unregistered [ 1863.235930] LNet: 51963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.246804] LNetError: 51963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.295845] LNet: Removed LNI 192.168.206.138@tcp [ 1864.892594] Key type .llcrypt unregistered [ 1864.895923] Key type ._llcrypt unregistered [ 1883.213713] Key type ._llcrypt registered [ 1883.215748] Key type .llcrypt registered [ 1883.298496] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1884.149635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.174689] alg: No test for adler32 (adler32-zlib) [ 1885.244605] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1885.477024] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1887.159430] Key type lgssc registered [ 1888.223610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.214463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.228805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.606443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.436590] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1906.154993] LustreError: 53357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.168619] LustreError: 53357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1906.856520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.137565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.061746] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1916.942984] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1921.244787] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1926.433721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.762540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.095183] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1939.699471] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1942.510203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1942.522531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.547843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.586473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.646732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.664708] Lustre: Skipped 1 previous similar message [ 1953.762709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.770835] Lustre: Skipped 1 previous similar message [ 1956.319538] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1956.499232] Lustre: server umount lustre-OST0000 complete [ 1963.490257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1963.497371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.508304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.517343] Lustre: Skipped 1 previous similar message [ 1965.551852] Lustre: server umount lustre-MDT0000 complete [ 1968.320137] LustreError: 53338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607108 with bad export cookie 15940476303229582188 [ 1968.320783] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.328187] LustreError: 53338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1968.645694] Lustre: server umount lustre-MDT0001 complete [ 1977.434708] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1979.618366] Key type lgssc unregistered [ 1979.866562] LNet: 56886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.877414] LNetError: 56886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.906244] LNet: Removed LNI 192.168.206.138@tcp [ 1980.565584] Key type .llcrypt unregistered [ 1980.571330] Key type ._llcrypt unregistered [ 1998.998859] Key type ._llcrypt registered [ 1999.002103] Key type .llcrypt registered [ 1999.086696] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1999.979399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.008161] alg: No test for adler32 (adler32-zlib) [ 2001.040961] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2001.255352] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2002.919204] Key type lgssc registered [ 2003.860828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.639730] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.671204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2026.271604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2031.391272] LustreError: 58252:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 2058.079651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.838487] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2069.926158] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2075.319283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.737991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.804598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2079.495578] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2080.739709] LustreError: 58277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.752420] LustreError: 58277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2085.864264] LustreError: 58278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.711870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.121711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.396420] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2097.465850] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2102.306978] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2106.341040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2106.353167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.373887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.877552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.881203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.889445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2107.914161] Lustre: Skipped 1 previous similar message [ 2110.140024] Lustre: server umount lustre-OST0000 complete [ 2132.450069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.455458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.462528] Lustre: Skipped 1 previous similar message [ 2132.471483] Lustre: Skipped 1 previous similar message [ 2137.261391] Lustre: server umount lustre-MDT0000 complete [ 2137.568072] LustreError: 59292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.590396] LustreError: 59292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2140.319352] LustreError: 58259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607280 with bad export cookie 362848408222693053 [ 2140.328995] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.337538] LustreError: 58259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2146.808533] Lustre: server umount lustre-MDT0001 complete [ 2156.055589] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2159.079613] Key type lgssc unregistered [ 2159.436676] LNet: 61994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.448596] LNetError: 61994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.460833] LNet: Removed LNI 192.168.206.138@tcp [ 2160.339392] Key type .llcrypt unregistered [ 2160.347181] Key type ._llcrypt unregistered [ 2179.305482] Key type ._llcrypt registered [ 2179.308545] Key type .llcrypt registered [ 2179.401498] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2180.112569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.172924] alg: No test for adler32 (adler32-zlib) [ 2181.267881] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2181.511977] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2183.178327] Key type lgssc registered [ 2184.217798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2190.744656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2190.765562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.136287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.132858] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2201.572382] LustreError: 63388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.589157] LustreError: 63388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2203.345966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.724562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.713531] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2213.443418] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2218.042523] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2223.091588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2223.441772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.609521] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2234.639990] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2236.795345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2252.256533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.271217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.283192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.285153] Lustre: Skipped 1 previous similar message [ 2252.300823] Lustre: Skipped 1 previous similar message [ 2256.817728] Lustre: server umount lustre-OST0000 complete [ 2262.498652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.501592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2262.511904] Lustre: Skipped 1 previous similar message [ 2266.530701] Lustre: server umount lustre-MDT0000 complete [ 2267.617880] LustreError: 63387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.626050] LustreError: 63387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2269.382445] LustreError: 63368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607409 with bad export cookie 2538912034584433812 [ 2269.386566] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2269.393031] LustreError: 63368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2269.686300] Lustre: server umount lustre-MDT0001 complete [ 2278.679696] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2281.008764] Key type lgssc unregistered [ 2281.237865] LNet: 66945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.249885] LNetError: 66945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.263520] LNet: Removed LNI 192.168.206.138@tcp [ 2281.900494] Key type .llcrypt unregistered [ 2281.902963] Key type ._llcrypt unregistered [ 2301.043225] Key type ._llcrypt registered [ 2301.046288] Key type .llcrypt registered [ 2301.107897] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2301.813183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.847244] alg: No test for adler32 (adler32-zlib) [ 2302.899449] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2303.122296] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2304.792390] Key type lgssc registered [ 2305.710736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.905500] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:57:32 (1781607452) [ 2317.875852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2317.917884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.247595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.262120] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2328.592840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.907391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.863154] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2336.918140] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2341.122472] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2346.217101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.450438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.075939] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2356.735591] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2364.911208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2371.063833] Lustre: Failing over lustre-MDT0000 [ 2371.306497] Lustre: server umount lustre-MDT0000 complete [ 2372.514628] LustreError: 68622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.529365] LustreError: 68622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2375.137490] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.139559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2375.147659] Lustre: Skipped 1 previous similar message [ 2375.148330] LustreError: 68627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.623095] LustreError: 70029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.640054] LustreError: 70029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2378.434752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.536839] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.676286] Lustre: *** cfs_fail_loc=135, val=0*** [ 2378.743479] LustreError: 71732:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2378.751495] LustreError: 71732:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.138@tcp: cfg command failed: rc = -2 [ 2378.760772] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2378.760772] [ 2378.773623] LustreError: MGC192.168.206.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2378.786933] LustreError: 71721:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2378.793574] LustreError: 71721:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2378.800484] LustreError: 71721:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2378.808296] LustreError: 71721:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2378.833210] Lustre: server umount lustre-MDT0000 complete [ 2378.836567] LustreError: 71721:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2380.260139] LustreError: 68622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.270058] LustreError: 68622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2384.268526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.386523] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.614761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.667574] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2387.861048] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2388.112890] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2389.993608] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2390.025583] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2390.056611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2394.592818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.600447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.611644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.115326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.120394] Lustre: Skipped 1 previous similar message [ 2399.810585] Lustre: server umount lustre-OST0000 complete [ 2405.344800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.345595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.349874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.352894] LustreError: Skipped 1 previous similar message [ 2405.369895] Lustre: Skipped 2 previous similar messages [ 2408.844034] Lustre: server umount lustre-MDT0000 complete [ 2410.464321] LustreError: 68627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.490920] LustreError: 68627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2411.320508] LustreError: 68609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607551 with bad export cookie 10399676778113722067 [ 2411.326753] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.332153] LustreError: 68609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2411.626719] Lustre: server umount lustre-MDT0001 complete [ 2419.048696] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2421.150227] Key type lgssc unregistered [ 2421.412843] LNet: 73508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.434767] LNetError: 73508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.449694] LNet: Removed LNI 192.168.206.138@tcp [ 2422.067649] Key type .llcrypt unregistered [ 2422.070036] Key type ._llcrypt unregistered [ 2438.460380] Key type ._llcrypt registered [ 2438.462616] Key type .llcrypt registered [ 2438.541990] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2439.372507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2439.395306] alg: No test for adler32 (adler32-zlib) [ 2440.363591] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2440.547359] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2442.231245] Key type lgssc registered [ 2443.052710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.247149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2448.265761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.651611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.758523] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2458.608283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.864234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2461.578843] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2465.720822] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2468.532278] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2471.823842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2472.014611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.848794] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2475.117762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2478.458820] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2487.431635] Lustre: Failing over lustre-MDT0000 [ 2487.684409] Lustre: server umount lustre-MDT0000 complete [ 2488.805875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.808081] LustreError: 74900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.819685] Lustre: Skipped 2 previous similar messages [ 2488.832813] LustreError: 74900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2491.297080] LustreError: 74897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.304734] LustreError: 74897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2493.818984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.896790] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.924058] LustreError: 74901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.933826] LustreError: 74901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2494.051770] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2494.147150] LustreError: 77985:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2494.157928] LustreError: 77985:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.138@tcp: cfg command failed: rc = -9 [ 2494.169932] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2494.169932] [ 2494.177682] LustreError: MGC192.168.206.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2494.188890] LustreError: 77974:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2494.194626] LustreError: 77974:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2494.201058] LustreError: 77974:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2494.206578] LustreError: 77974:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2494.234098] Lustre: server umount lustre-MDT0000 complete [ 2494.236956] LustreError: 77974:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2496.409244] LustreError: 74895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.908967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.990302] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2499.241457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.290603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2501.521397] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2502.342962] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2504.684125] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2504.714938] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2504.745342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2509.280404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2509.288886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.301479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.793516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2513.528360] Lustre: server umount lustre-OST0000 complete [ 2520.033708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2520.034826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.050563] Lustre: Skipped 1 previous similar message [ 2520.055505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.059478] Lustre: Skipped 2 previous similar messages [ 2525.152862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2525.157415] Lustre: Skipped 1 previous similar message [ 2529.250140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.259436] Lustre: Skipped 1 previous similar message [ 2530.271140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2530.392116] Lustre: server umount lustre-MDT0000 complete [ 2532.908945] LustreError: 74882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607672 with bad export cookie 13497813960393923303 [ 2532.914614] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.927070] LustreError: 74882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2533.130819] Lustre: server umount lustre-MDT0001 complete [ 2539.529146] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2541.626155] Key type lgssc unregistered [ 2541.841955] LNet: 79777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2541.848811] LNetError: 79777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.886381] LNet: Removed LNI 192.168.206.138@tcp [ 2543.433821] Key type .llcrypt unregistered [ 2543.436129] Key type ._llcrypt unregistered [ 2551.586229] Key type ._llcrypt registered [ 2551.589647] Key type .llcrypt registered [ 2551.654585] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2551.840300] Key type .llcrypt unregistered [ 2551.842108] Key type ._llcrypt unregistered [ 2566.247727] Key type ._llcrypt registered [ 2566.249593] Key type .llcrypt registered [ 2566.322741] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2566.964103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.985452] alg: No test for adler32 (adler32-zlib) [ 2567.925431] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2568.101929] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2569.719218] Key type lgssc registered [ 2570.354548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2574.954596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2574.969488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.248181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.297344] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2584.264296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.481138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2586.670908] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2590.566833] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2593.491096] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2597.491732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.711453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.101832] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2601.835933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2605.540923] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2615.270157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.279618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.280285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.287978] Lustre: Skipped 1 previous similar message [ 2620.385308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2620.389074] Lustre: Skipped 1 previous similar message [ 2620.809757] Lustre: server umount lustre-OST0000 complete [ 2625.505419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2625.505797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.510843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.523829] Lustre: Skipped 2 previous similar messages [ 2625.536196] Lustre: Skipped 1 previous similar message [ 2630.624392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.627978] Lustre: Skipped 1 previous similar message [ 2635.744920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.748893] Lustre: Skipped 1 previous similar message [ 2637.279211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2637.372111] Lustre: server umount lustre-MDT0000 complete [ 2639.564242] LustreError: 81639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607779 with bad export cookie 16404712748618076993 [ 2639.566816] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.570840] LustreError: 81639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2639.757036] Lustre: server umount lustre-MDT0001 complete [ 2645.366848] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2646.905334] Key type lgssc unregistered [ 2647.095724] LNet: 85187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2647.099957] LNetError: 85187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.111479] LNet: Removed LNI 192.168.206.138@tcp [ 2647.590382] Key type .llcrypt unregistered [ 2647.592596] Key type ._llcrypt unregistered [ 2660.101109] Key type ._llcrypt registered [ 2660.102954] Key type .llcrypt registered [ 2660.162438] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2660.667096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2660.690069] alg: No test for adler32 (adler32-zlib) [ 2661.616795] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2661.743334] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2663.359557] Key type lgssc registered [ 2663.929520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2667.919087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2667.928887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.216985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.338805] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2676.070218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.259960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2678.654313] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2682.680248] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2685.521582] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2689.277574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.471641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.653197] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2694.569801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2696.649953] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2706.916177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2706.921273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2706.930849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.758807] Lustre: server umount lustre-OST0000 complete [ 2717.152636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.154567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2717.160108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2717.160121] Lustre: Skipped 1 previous similar message [ 2717.161771] Lustre: Skipped 2 previous similar messages [ 2720.230763] Lustre: server umount lustre-MDT0000 complete [ 2722.274220] LustreError: 87305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.280606] LustreError: 87305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2722.404312] LustreError: 86558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607862 with bad export cookie 481487498456137210 [ 2722.408964] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.410275] LustreError: 86558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2722.570538] Lustre: server umount lustre-MDT0001 complete [ 2727.863940] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2729.292126] Key type lgssc unregistered [ 2729.451440] LNet: 90105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.455642] LNetError: 90105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.464441] LNet: Removed LNI 192.168.206.138@tcp [ 2729.879650] Key type .llcrypt unregistered [ 2729.881351] Key type ._llcrypt unregistered [ 2742.329983] Key type ._llcrypt registered [ 2742.332018] Key type .llcrypt registered [ 2742.399475] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2742.964280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2742.975154] alg: No test for adler32 (adler32-zlib) [ 2743.883196] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2744.028615] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2745.655199] Key type lgssc registered [ 2746.306840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2750.558900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2750.573536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.846608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.982611] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2758.917113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.129556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.227531] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2764.771510] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2767.249985] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2770.113170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.286762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.119266] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2776.425248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2776.472685] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2788.834040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.841973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2788.843273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2788.850349] Lustre: Skipped 1 previous similar message [ 2793.141016] Lustre: server umount lustre-OST0000 complete [ 2794.976330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2794.981442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.989337] Lustre: Skipped 1 previous similar message [ 2794.993256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.072971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.073526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.192816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.196563] Lustre: Skipped 2 previous similar messages [ 2808.288531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.290869] Lustre: Skipped 1 previous similar message [ 2809.311291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2809.391277] Lustre: server umount lustre-MDT0000 complete [ 2810.931424] LustreError: 91475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607950 with bad export cookie 14828285744239035394 [ 2810.934516] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.936019] LustreError: 91475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2811.073389] Lustre: server umount lustre-MDT0001 complete [ 2815.378874] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2816.650629] Key type lgssc unregistered [ 2816.797147] LNet: 95027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.800610] LNetError: 95027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2816.810813] LNet: Removed LNI 192.168.206.138@tcp [ 2817.159401] Key type .llcrypt unregistered [ 2817.160834] Key type ._llcrypt unregistered [ 2827.027637] Key type ._llcrypt registered [ 2827.029290] Key type .llcrypt registered [ 2827.075615] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2827.500902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2827.521372] alg: No test for adler32 (adler32-zlib) [ 2828.400586] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2828.494733] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2830.087173] Key type lgssc registered [ 2830.566069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2834.100126] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2834.113456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.567319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2854.687212] LustreError: 96392:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 2881.312266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.222346] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2888.007892] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2890.936636] Lustre: server umount lustre-OST0000 complete [ 2895.963804] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:07:15 (1781608035) [ 2900.124841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.404649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.946625] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2905.568782] LustreError: 98028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.584450] LustreError: 98028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2908.090766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.297689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.005394] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2915.333376] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2918.454104] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.961851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.154769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.001704] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2925.224860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2928.982851] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2938.850371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.852185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.855668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.861237] Lustre: Skipped 1 previous similar message [ 2938.867801] Lustre: Skipped 1 previous similar message [ 2940.446373] Lustre: server umount lustre-OST0000 complete [ 2943.968750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.971859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2943.974829] Lustre: Skipped 1 previous similar message [ 2944.371635] Lustre: server umount lustre-MDT0000 complete [ 2946.008440] LustreError: 98009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608085 with bad export cookie 4586834572009288740 [ 2946.013396] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.015977] LustreError: 98009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2946.153853] Lustre: server umount lustre-MDT0001 complete [ 2951.286900] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2952.624731] Key type lgssc unregistered [ 2952.804476] LNet: 101715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.810525] LNetError: 101715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.825593] LNet: Removed LNI 192.168.206.138@tcp [ 2953.262349] Key type .llcrypt unregistered [ 2953.265589] Key type ._llcrypt unregistered [ 2972.360975] Key type ._llcrypt registered [ 2972.362330] Key type .llcrypt registered [ 2972.406963] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2972.967186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.981744] alg: No test for adler32 (adler32-zlib) [ 2973.890341] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2974.005877] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2975.615273] Key type lgssc registered [ 2976.127470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2979.654476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2979.665238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.915111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.664168] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2986.588371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.791322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.745216] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2992.099960] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2994.658686] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2997.413241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.558125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.951090] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3002.645001] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3005.738094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3016.160369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3016.163628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.169238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.185925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.186098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.187011] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3017.197350] Lustre: Skipped 1 previous similar message [ 3021.661740] Lustre: server umount lustre-OST0000 complete [ 3027.424813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.425766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3027.430469] Lustre: Skipped 1 previous similar message [ 3027.435408] Lustre: Skipped 1 previous similar message [ 3029.340640] Lustre: server umount lustre-MDT0000 complete [ 3030.745465] LustreError: 105718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608170 with bad export cookie 13575619063392014432 [ 3030.748374] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.750065] LustreError: 105718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3030.876915] Lustre: server umount lustre-MDT0001 complete [ 3034.928790] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3036.133390] Key type lgssc unregistered [ 3036.293816] LNet: 107427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.297391] LNetError: 107427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.306548] LNet: Removed LNI 192.168.206.138@tcp [ 3036.691513] Key type .llcrypt unregistered [ 3036.693270] Key type ._llcrypt unregistered [ 3044.972934] Key type ._llcrypt registered [ 3044.974344] Key type .llcrypt registered [ 3045.020500] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3045.473476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3045.534169] alg: No test for adler32 (adler32-zlib) [ 3046.403066] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3046.497630] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3048.095134] Key type lgssc registered [ 3048.495751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.446651] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3051.453839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.911492] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3072.031271] LustreError: 108646:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 3098.655283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.220775] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3104.394524] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3107.422748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.571043] LustreError: MGC192.168.206.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3107.578461] LustreError: 109669:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3107.582062] LustreError: 109669:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3107.585564] LustreError: 109669:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3107.627037] Lustre: server umount lustre-MDT0000 complete [ 3107.629220] LustreError: 109669:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3121.170275] Lustre: server umount lustre-OST0000 complete [ 3125.353428] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3129.723926] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3145.655499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.792144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3145.811459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3145.858168] Lustre: lustre-MDT0000: new disk, initializing [ 3145.898919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.907625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3147.876378] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3153.540524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.591910] Lustre: 112456:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3153.629439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3153.631837] Lustre: Skipped 1 previous similar message [ 3153.675981] Lustre: lustre-MDT0001: new disk, initializing [ 3153.727496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3153.748543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3153.758636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3155.725412] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3160.351378] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3162.486835] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3165.140077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3165.263400] Lustre: lustre-OST0000: new disk, initializing [ 3165.266425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3165.271039] Lustre: 114425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3165.323457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.750514] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3172.070596] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3173.795326] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3173.887770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3173.892117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3173.933153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3174.910787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3176.674260] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3176.789235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3178.979636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3178.981791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.982685] LustreError: Skipped 1 previous similar message [ 3178.994025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3178.997273] Lustre: Skipped 1 previous similar message [ 3183.774146] Lustre: server umount lustre-OST0000 complete [ 3189.217070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.218179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.223346] Lustre: Skipped 2 previous similar messages [ 3189.227940] Lustre: Skipped 1 previous similar message [ 3191.484609] Lustre: server umount lustre-MDT0000 complete [ 3193.113712] LustreError: 112449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608332 with bad export cookie 6958952463070127533 [ 3193.118388] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.118831] LustreError: 112449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3193.260325] Lustre: server umount lustre-MDT0001 complete [ 3196.602583] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:12:16 (1781608336) [ 3198.204632] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3205.449121] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3209.520621] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3226.388833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.529388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3226.548058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3226.598067] Lustre: lustre-MDT0000: new disk, initializing [ 3226.650683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.661910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3228.893799] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3234.894262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.975832] Lustre: 119323:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3235.013164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3235.015833] Lustre: Skipped 1 previous similar message [ 3235.056512] Lustre: lustre-MDT0001: new disk, initializing [ 3235.104848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3235.123395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3235.128244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3236.828996] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3241.783954] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3244.722651] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3248.468876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.649380] Lustre: lustre-OST0000: new disk, initializing [ 3248.651685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3248.655792] Lustre: 121293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3250.500865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3250.509293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3250.568245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3252.057694] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3257.441778] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3259.574878] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3259.678122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3261.899791] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3262.011301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3266.022989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.023945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.030272] LustreError: Skipped 1 previous similar message [ 3266.037395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.047399] Lustre: Skipped 1 previous similar message [ 3271.141086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.143759] Lustre: Skipped 1 previous similar message [ 3276.262128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.265568] Lustre: Skipped 1 previous similar message [ 3277.279186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3277.361912] Lustre: server umount lustre-OST0000 complete [ 3281.381169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.381385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3281.388468] Lustre: Skipped 2 previous similar messages [ 3285.300303] Lustre: server umount lustre-MDT0000 complete [ 3286.970356] LustreError: 119317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608426 with bad export cookie 6958952463070128401 [ 3286.974404] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.980536] LustreError: 119317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3287.123964] Lustre: server umount lustre-MDT0001 complete [ 3290.831191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.089129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.094450] Lustre: Skipped 1 previous similar message [ 3292.894811] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3296.225408] LustreError: 123508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.244858] LustreError: 123508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3297.132464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.102065] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3302.389956] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3304.779542] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3307.988128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.806025] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3314.058595] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3327.389699] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3331.040288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.042265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.044569] LustreError: Skipped 1 previous similar message [ 3331.050065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3331.053565] Lustre: Skipped 2 previous similar messages [ 3343.328684] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3343.398935] Lustre: server umount lustre-OST0000 complete [ 3348.450322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.450388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3348.451065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.451071] Lustre: Skipped 6 previous similar messages [ 3348.463488] Lustre: Skipped 2 previous similar messages [ 3348.467314] LustreError: Skipped 1 previous similar message [ 3351.218767] Lustre: server umount lustre-MDT0000 complete [ 3353.246195] LustreError: 123487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608493 with bad export cookie 6958952463070129283 [ 3353.250929] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3353.254306] LustreError: 123487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3353.464902] Lustre: server umount lustre-MDT0001 complete [ 3358.938829] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3360.477996] Key type lgssc unregistered [ 3360.628683] LNet: 127153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3360.635770] LNetError: 127153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3360.644565] LNet: Removed LNI 192.168.206.138@tcp [ 3361.039643] Key type .llcrypt unregistered [ 3361.041290] Key type ._llcrypt unregistered [ 3368.846176] Key type ._llcrypt registered [ 3368.847649] Key type .llcrypt registered [ 3368.897992] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3376.124965] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3376.461995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3376.528489] alg: No test for adler32 (adler32-zlib) [ 3377.404896] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3377.490539] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3379.079182] Key type lgssc registered [ 3379.450803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3391.428923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3391.436529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.540984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3392.558233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3392.594170] Lustre: lustre-MDT0000: new disk, initializing [ 3392.632534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.642126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3394.093286] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3398.763830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.818204] Lustre: 130279:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3398.851485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3398.854221] Lustre: Skipped 1 previous similar message [ 3398.895494] Lustre: lustre-MDT0001: new disk, initializing [ 3398.942275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3398.956430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3398.963323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3400.652885] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3404.902250] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3407.201217] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3410.126673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.247316] Lustre: lustre-OST0000: new disk, initializing [ 3410.249645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3410.253328] Lustre: 132244:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3410.288217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.647751] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3417.033948] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3418.774707] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3419.644749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3419.649171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3419.693682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3419.888744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3421.630444] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3421.737429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3424.739353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.739437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.743105] LustreError: Skipped 1 previous similar message [ 3424.747991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.751185] Lustre: Skipped 1 previous similar message [ 3428.629684] Lustre: server umount lustre-OST0000 complete [ 3434.977047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.977114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3434.977705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3434.980818] Lustre: Skipped 2 previous similar messages [ 3440.096504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.099248] Lustre: Skipped 2 previous similar messages [ 3444.191181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3444.193642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.198451] Lustre: Skipped 1 previous similar message [ 3444.291633] Lustre: server umount lustre-MDT0000 complete [ 3445.729392] LustreError: 130271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608585 with bad export cookie 7814930511849521246 [ 3445.732782] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.735029] LustreError: 130271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3445.858741] Lustre: server umount lustre-MDT0001 complete [ 3449.094722] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:16:29 (1781608589) [ 3451.580232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.764233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.193145] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3456.199247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.324605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3457.672524] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3460.168789] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3462.074472] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3466.721459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3466.721468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.722424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.731138] Lustre: Skipped 1 previous similar message [ 3468.892174] Lustre: server umount lustre-MDT0000 complete [ 3470.083917] LustreError: 134643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608609 with bad export cookie 7814930511849522114 [ 3470.086650] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3470.087205] LustreError: 134643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3470.203036] Lustre: server umount lustre-MDT0001 complete [ 3473.040485] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:16:53 (1781608613) [ 3475.320193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.783583] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3495.903505] LustreError: 137076:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 3521.503411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.431279] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3526.047462] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3526.783700] Lustre: server umount lustre-OST0000 complete [ 3529.533037] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:17:49 (1781608669) [ 3531.712536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.868770] LustreError: 138530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.877572] LustreError: 138530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3533.262399] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3536.225015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.740564] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3540.368760] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3542.459977] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3544.981804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.105020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.108640] Lustre: Skipped 2 previous similar messages [ 3547.070599] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3549.560480] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3561.951776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3561.954048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.958931] Lustre: Skipped 1 previous similar message [ 3561.961505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.964106] Lustre: Skipped 2 previous similar messages [ 3566.561623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.565292] Lustre: Skipped 1 previous similar message [ 3567.961789] Lustre: server umount lustre-MDT0000 complete [ 3569.082031] LustreError: 138512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608708 with bad export cookie 7814930511849522765 [ 3569.084527] LustreError: 138512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3569.085409] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3569.191221] Lustre: server umount lustre-MDT0001 complete [ 3580.505764] Lustre: server umount lustre-OST0000 complete [ 3583.109758] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:18:43 (1781608723) [ 3585.225789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.391460] LustreError: 142200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.417083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.730294] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3589.444329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.867926] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3593.129868] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3594.909540] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3597.123384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.072370] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3601.366554] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3602.723071] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3610.538160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3610.950520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3612.397829] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3612.473256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3614.687846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3614.690414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.695703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.698324] Lustre: Skipped 3 previous similar messages [ 3619.215460] Lustre: server umount lustre-OST0000 complete [ 3625.439762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3625.442059] LustreError: Skipped 1 previous similar message [ 3626.583412] Lustre: server umount lustre-MDT0000 complete [ 3627.820289] LustreError: 142180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608767 with bad export cookie 7814930511849523822 [ 3627.822758] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.824680] LustreError: 142180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3627.939584] Lustre: server umount lustre-MDT0001 complete [ 3630.748418] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:19:31 (1781608771) [ 3633.120331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3648.543702] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3653.663530] LustreError: 146413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 3679.263252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.265428] Lustre: Skipped 2 previous similar messages [ 3681.201777] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3683.570577] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3685.810853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.955321] LustreError: 146438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.001346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3687.296934] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3690.096154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.639791] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3694.242854] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3696.186517] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3697.637587] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3697.728602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3699.196495] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3699.292355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3700.704602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3700.709160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.715290] Lustre: Skipped 3 previous similar messages [ 3700.718437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.720965] Lustre: Skipped 5 previous similar messages [ 3706.141252] Lustre: server umount lustre-MDT0000 complete [ 3707.646429] LustreError: 146419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608847 with bad export cookie 7814930511849524830 [ 3707.650584] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.652129] LustreError: 146419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3707.773156] Lustre: server umount lustre-MDT0001 complete [ 3719.452663] Lustre: server umount lustre-OST0000 complete [ 3722.447838] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:21:02 (1781608862) [ 3724.965577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.447432] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3745.567416] LustreError: 150603:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 3773.205052] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3775.702224] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3778.020295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.161472] LustreError: 150628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.222073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3779.717699] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3782.763326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.325335] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3786.937824] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3788.691467] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3790.888081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3790.961748] Lustre: lustre-OST0001: new disk, initializing [ 3790.964216] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3790.967664] Lustre: 153471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3793.001874] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3796.712796] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3798.115780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3801.084205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3801.087831] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3801.118965] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3801.236044] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3802.585956] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3802.657830] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3803.616113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.619702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.624798] Lustre: Skipped 2 previous similar messages [ 3803.627153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.629065] Lustre: Skipped 3 previous similar messages [ 3809.400086] Lustre: server umount lustre-OST0000 complete [ 3811.298670] LustreError: 150619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.305706] LustreError: 150619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3816.716174] Lustre: server umount lustre-OST0001 complete [ 3832.287151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3832.344949] Lustre: server umount lustre-MDT0000 complete [ 3833.509275] LustreError: 153470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608973 with bad export cookie 7814930511849525782 [ 3833.513126] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.513499] LustreError: 153470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3833.621735] Lustre: server umount lustre-MDT0001 complete [ 3841.075663] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:23:01 (1781608981) [ 3841.612807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3842.136802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3842.660659] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:23:03 (1781608983) [ 3844.879330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.966442] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3844.976733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3845.047587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.049827] Lustre: Skipped 6 previous similar messages [ 3846.399331] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3849.200774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.234047] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3849.243340] Lustre: 157051:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3850.692214] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3852.951928] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3854.662053] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3856.797710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3856.860762] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3858.840104] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3859.670135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3861.143326] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3862.444544] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3862.506557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3863.835982] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3863.908311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3866.581801] Lustre: server umount lustre-OST0000 complete [ 3880.496207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3882.593051] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3885.013241] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3885.730774] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3885.736614] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3885.736777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3886.575103] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3886.657952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3888.017771] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3888.089677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3889.807990] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff916a1a35b000.ost_server_uuid 50 [ 3890.285899] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff916a1a35b000.ost_server_uuid in FULL state after 0 sec [ 3890.341741] LustreError: 158916:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.38@tcp) returned error from blocking AST (req@000000000ddd51bf x1868151932510208 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000def72402/0x6c7435e52f271b62 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.38@tcp remote: 0x4ca05620fdd71c72 expref: 5 pid: 157057 timeout: 3990 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3890.354278] LustreError: lustre-MDT0000: A client on nid 192.168.206.38@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3890.358533] LustreError: 157047:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.38@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000def72402/0x6c7435e52f271b62 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.38@tcp remote: 0x4ca05620fdd71c72 expref: 6 pid: 157057 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3895.777137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3895.779535] LustreError: Skipped 5 previous similar messages [ 3895.781591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.785835] Lustre: Skipped 7 previous similar messages [ 3897.482080] Lustre: server umount lustre-OST0000 complete [ 3913.183114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3913.247474] Lustre: server umount lustre-MDT0000 complete [ 3914.448282] LustreError: 162113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609054 with bad export cookie 7814930511849526965 [ 3914.451213] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3914.452121] LustreError: 162113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3920.628806] Lustre: server umount lustre-MDT0001 complete [ 3923.943063] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3924.916696] Key type lgssc unregistered [ 3925.041377] LNet: 163153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3925.045237] LNetError: 163153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3925.053283] LNet: Removed LNI 192.168.206.138@tcp [ 3925.325351] Key type .llcrypt unregistered [ 3925.326867] Key type ._llcrypt unregistered [ 3933.740441] Key type ._llcrypt registered [ 3933.741758] Key type .llcrypt registered [ 3933.781359] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3934.081609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3934.109231] alg: No test for adler32 (adler32-zlib) [ 3934.966631] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3935.054282] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3936.639159] Key type lgssc registered [ 3937.049527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3939.772649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3939.780588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.949412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.355966] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3945.189715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.319938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3946.677968] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3949.137336] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.935629] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3953.236187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3953.352690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.343167] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3957.840358] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3965.673491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3970.520453] Lustre: Failing over lustre-MDT0000 [ 3970.665783] Lustre: server umount lustre-MDT0000 complete [ 3971.041977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.042980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3971.043207] LustreError: 164537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.043221] LustreError: 164537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3971.047054] Lustre: Skipped 2 previous similar messages [ 3976.160933] LustreError: 165282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.166608] LustreError: 165282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3981.280893] LustreError: 164543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.287051] LustreError: 164543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3981.527017] LustreError: 167140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.38@tcp arrived at 1781609121 with bad export cookie 5643385098794206531 [ 3981.531203] LustreError: 167140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3983.859337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.900951] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.993410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.011956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3985.272538] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3985.974104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3985.977874] Lustre: lustre-MDT0000: Denying connection for new client 72474e07-54c9-4ede-8da6-2e082bd683da (at 192.168.206.38@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3989.477384] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3994.594931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.597137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.598913] Lustre: Skipped 1 previous similar message [ 3999.714434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.716069] Lustre: Skipped 1 previous similar message [ 4004.834546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.837329] Lustre: Skipped 2 previous similar messages [ 4006.879118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4006.936434] Lustre: server umount lustre-OST0000 complete [ 4008.163338] LustreError: 168289:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4008.167512] Lustre: 167638:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4008.170412] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4008.176666] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4008.180730] LustreError: 167638:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e6d71dda x1868152516414848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4008.186613] LustreError: 167638:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4008.189834] LustreError: 167638:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4008.193132] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 4008.294288] Lustre: server umount lustre-MDT0000 complete [ 4009.527116] LustreError: 165276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609149 with bad export cookie 5643385098794206860 [ 4009.531926] LustreError: 163853:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4009.532047] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.532241] LustreError: 165276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4015.735391] Lustre: server umount lustre-MDT0001 complete [ 4019.241129] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4020.276553] Key type lgssc unregistered [ 4020.422305] LNet: 169039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4020.424975] LNetError: 169039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4020.433333] LNet: Removed LNI 192.168.206.138@tcp [ 4020.740232] Key type .llcrypt unregistered [ 4020.741616] Key type ._llcrypt unregistered [ 4029.956189] Key type ._llcrypt registered [ 4029.957519] Key type .llcrypt registered [ 4029.994274] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4030.305600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4030.338034] alg: No test for adler32 (adler32-zlib) [ 4031.191358] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4031.277158] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4032.855101] Key type lgssc registered [ 4033.211543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4035.807486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4035.814973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.971599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.358618] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4041.244896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.390409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4042.768866] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4045.169370] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4046.993844] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4049.270209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4049.409223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.455075] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4053.985806] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4061.731842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4066.786343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.786758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.791422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.794408] LustreError: Skipped 1 previous similar message [ 4066.797951] Lustre: Skipped 1 previous similar message [ 4067.982697] Lustre: server umount lustre-OST0000 complete [ 4071.325878] Lustre: server umount lustre-MDT0000 complete [ 4071.905774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.906483] LustreError: 170425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.911418] Lustre: Skipped 2 previous similar messages [ 4071.920142] LustreError: 170425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4072.571393] LustreError: 170454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609212 with bad export cookie 5534802971771706600 [ 4072.575427] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.577919] LustreError: 170454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4072.691022] Lustre: server umount lustre-MDT0001 complete [ 4076.376411] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4077.450105] Key type lgssc unregistered [ 4077.585351] LNet: 173978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4077.588603] LNetError: 173978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4077.597463] LNet: Removed LNI 192.168.206.138@tcp [ 4077.922474] Key type .llcrypt unregistered [ 4077.924444] Key type ._llcrypt unregistered [ 4089.550525] Key type ._llcrypt registered [ 4089.551771] Key type .llcrypt registered [ 4089.589389] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4089.986265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4090.018433] alg: No test for adler32 (adler32-zlib) [ 4090.877800] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4090.965079] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4092.543104] Key type lgssc registered [ 4092.923159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4095.574025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4095.579805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.738723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.148540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4100.989034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.100550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4102.424716] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4104.829503] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4106.618738] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.720651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4108.827566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4110.590733] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4112.837416] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4122.148932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4126.175696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4126.178369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.182677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.201049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.201416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.202642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4130.834168] Lustre: server umount lustre-OST0000 complete [ 4132.320338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.321131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4132.325270] Lustre: Skipped 1 previous similar message [ 4132.328131] Lustre: Skipped 1 previous similar message [ 4137.439890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.441934] Lustre: Skipped 1 previous similar message [ 4138.134417] Lustre: server umount lustre-MDT0000 complete [ 4139.285110] LustreError: 175637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609279 with bad export cookie 10734281295903015645 [ 4139.288035] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.288602] LustreError: 175637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4139.390814] Lustre: server umount lustre-MDT0001 complete [ 4142.685610] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4143.610595] Key type lgssc unregistered [ 4143.734312] LNet: 179208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4143.736775] LNetError: 179208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4143.749249] LNet: Removed LNI 192.168.206.138@tcp [ 4144.000996] Key type .llcrypt unregistered [ 4144.002076] Key type ._llcrypt unregistered [ 4151.884738] Key type ._llcrypt registered [ 4151.885980] Key type .llcrypt registered [ 4151.921968] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4152.289244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4152.298573] alg: No test for adler32 (adler32-zlib) [ 4153.135441] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4153.204672] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4154.783151] Key type lgssc registered [ 4155.085311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4158.255691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4158.262250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.380592] Lustre: *** cfs_fail_loc=135, val=0*** [ 4159.417586] LustreError: 180706:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4159.420132] LustreError: 180706:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.138@tcp: cfg command failed: rc = -2 [ 4159.422919] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4159.422919] [ 4159.425527] LustreError: MGC192.168.206.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4159.429934] LustreError: 180668:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4159.432383] LustreError: 180668:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4159.434608] LustreError: 180668:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 4159.436568] LustreError: 180668:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4159.470518] Lustre: server umount lustre-MDT0000 complete [ 4159.471805] LustreError: 180668:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4161.527159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.689986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.943692] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4165.563791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.673858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4166.932201] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4169.043098] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4170.622540] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4175.328624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.334488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.336203] Lustre: Skipped 1 previous similar message [ 4177.368104] Lustre: server umount lustre-MDT0000 complete [ 4178.569270] LustreError: 181058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609318 with bad export cookie 2322207528878581789 [ 4178.572869] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.573021] LustreError: 181058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4178.689022] Lustre: server umount lustre-MDT0001 complete [ 4182.339478] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4183.375029] Key type lgssc unregistered [ 4183.495367] LNet: 183396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4183.497662] LNetError: 183396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4183.504281] LNet: Removed LNI 192.168.206.138@tcp [ 4183.782370] Key type .llcrypt unregistered [ 4183.783788] Key type ._llcrypt unregistered [ 4190.188453] Key type ._llcrypt registered [ 4190.189603] Key type .llcrypt registered [ 4190.224512] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4190.322339] Key type .llcrypt unregistered [ 4190.323455] Key type ._llcrypt unregistered [ 4197.318810] Key type ._llcrypt registered [ 4197.319807] Key type .llcrypt registered [ 4197.359227] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4197.746327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4197.759096] alg: No test for adler32 (adler32-zlib) [ 4198.626302] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4198.708511] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4200.295182] Key type lgssc registered [ 4200.678772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4203.331805] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4203.339855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4218.783288] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4223.903160] LustreError: 185503:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 4250.527142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.395339] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4254.745054] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4257.057665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.234410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4257.262121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4258.607698] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4261.495878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.638106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4263.109950] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4265.616682] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4267.422342] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4268.400408] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4276.195149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.200717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.202504] Lustre: Skipped 1 previous similar message [ 4279.371619] Lustre: server umount lustre-OST0000 complete [ 4281.312680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.313304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.316989] Lustre: Skipped 2 previous similar messages [ 4286.432085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.434053] Lustre: Skipped 2 previous similar messages [ 4286.613604] Lustre: server umount lustre-MDT0000 complete [ 4287.828448] LustreError: 188162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609427 with bad export cookie 211026749589778361 [ 4287.835486] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4287.963133] Lustre: server umount lustre-MDT0001 complete [ 4291.287206] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4292.216181] Key type lgssc unregistered [ 4292.326349] LNet: 189414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4292.328891] LNetError: 189414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4292.337365] LNet: Removed LNI 192.168.206.138@tcp [ 4292.599820] Key type .llcrypt unregistered [ 4292.601521] Key type ._llcrypt unregistered [ 4301.045320] Key type ._llcrypt registered [ 4301.046877] Key type .llcrypt registered [ 4301.081744] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4301.375398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4301.390736] alg: No test for adler32 (adler32-zlib) [ 4302.226926] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4302.302716] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4303.879152] Key type lgssc registered [ 4304.219592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4306.601594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4306.608791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.759339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.020331] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4311.683852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.803330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4313.166919] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4315.402399] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.953735] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4318.998612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.099230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.778922] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4322.947407] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4333.479223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4339.814598] Lustre: Failing over lustre-MDT0000 [ 4339.914357] Lustre: server umount lustre-MDT0000 complete [ 4342.751810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4342.754679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.759798] LustreError: 190803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.764937] LustreError: 190803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4343.702116] LustreError: 190798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.778081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.783112] Lustre: Skipped 1 previous similar message [ 4348.816152] LustreError: 190798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.822161] LustreError: 190798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4352.818891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.867316] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4352.980529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.012524] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4353.937578] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4354.389783] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4355.965585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4358.117844] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4358.123689] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4358.141889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4359.225583] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4362.207914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4362.212142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.218485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4363.234891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.234960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4363.240708] Lustre: Skipped 1 previous similar message [ 4367.433424] Lustre: server umount lustre-OST0000 complete [ 4373.472814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4373.472833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.473525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.480976] Lustre: Skipped 2 previous similar messages [ 4374.810979] Lustre: server umount lustre-MDT0000 complete [ 4376.033319] LustreError: 190782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609515 with bad export cookie 11799379821976202464 [ 4376.037056] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.037344] LustreError: 190782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4376.159722] Lustre: server umount lustre-MDT0001 complete [ 4379.843830] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4380.952276] Key type lgssc unregistered [ 4381.084353] LNet: 195649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4381.087416] LNetError: 195649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4381.096376] LNet: Removed LNI 192.168.206.138@tcp [ 4381.413193] Key type .llcrypt unregistered [ 4381.414411] Key type ._llcrypt unregistered [ 4389.901975] Key type ._llcrypt registered [ 4389.903026] Key type .llcrypt registered [ 4389.944922] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4390.277182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4390.285670] alg: No test for adler32 (adler32-zlib) [ 4391.154065] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4391.243449] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4392.831188] Key type lgssc registered [ 4393.163688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4395.464120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4395.470807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.608491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.856397] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4400.428272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.537504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4401.757379] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4403.832619] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4405.370111] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4407.260139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.357301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.048449] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4411.079418] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4421.731339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4427.256585] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4438.011214] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4444.342031] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4456.928721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4456.931245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.935122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4457.954068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4457.954137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.954456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.956894] Lustre: Skipped 1 previous similar message [ 4459.463907] Lustre: server umount lustre-OST0000 complete [ 4462.047762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.049879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.053784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4466.708969] Lustre: server umount lustre-MDT0000 complete [ 4467.786297] LustreError: 197018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609607 with bad export cookie 15633758766090587154 [ 4467.788879] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.789891] LustreError: 197018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4467.901381] Lustre: server umount lustre-MDT0001 complete [ 4471.307076] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4472.300407] Key type lgssc unregistered [ 4472.433333] LNet: 200929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4472.435821] LNetError: 200929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4472.445286] LNet: Removed LNI 192.168.206.138@tcp [ 4472.737759] Key type .llcrypt unregistered [ 4472.739026] Key type ._llcrypt unregistered [ 4480.008868] Key type ._llcrypt registered [ 4480.010905] Key type .llcrypt registered [ 4480.053932] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4480.513088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4480.521034] alg: No test for adler32 (adler32-zlib) [ 4481.387580] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4481.476065] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4483.063154] Key type lgssc registered [ 4483.441161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4486.073474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4486.081891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.258284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.695575] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4491.592583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.742413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4493.110818] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4495.499900] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4497.273208] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4499.581328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4499.704539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.649124] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4503.991821] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4512.994772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4516.137292] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4523.796741] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4533.071243] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4541.945819] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4549.089798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4549.090753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.096291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4549.097876] Lustre: Skipped 1 previous similar message [ 4554.209036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4554.211761] Lustre: Skipped 1 previous similar message [ 4554.514368] Lustre: server umount lustre-OST0000 complete [ 4558.303917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4558.307652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.313785] Lustre: Skipped 1 previous similar message [ 4558.316417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4559.329265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.448280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.451282] Lustre: Skipped 3 previous similar messages [ 4569.568022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4569.569881] Lustre: Skipped 1 previous similar message [ 4570.079105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4570.140083] Lustre: server umount lustre-MDT0000 complete [ 4571.374610] LustreError: 204869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609711 with bad export cookie 12908134344416059067 [ 4571.378637] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.379331] LustreError: 204869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4571.504630] Lustre: server umount lustre-MDT0001 complete [ 4575.088597] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4576.227834] Key type lgssc unregistered [ 4576.377575] LNet: 207640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4576.380545] LNetError: 207640:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4576.392493] LNet: Removed LNI 192.168.206.138@tcp [ 4576.710299] Key type .llcrypt unregistered [ 4576.711970] Key type ._llcrypt unregistered [ 4586.023894] Key type ._llcrypt registered [ 4586.025133] Key type .llcrypt registered [ 4586.065163] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4586.377031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4586.437062] alg: No test for adler32 (adler32-zlib) [ 4587.293642] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4587.380284] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4588.967113] Key type lgssc registered [ 4589.363612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4592.130638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4592.138556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.336638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4594.827793] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4597.667406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.804725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4599.232227] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4601.797130] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4603.754755] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4606.118366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.245884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.282891] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4610.737904] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4618.533734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4627.935742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4627.939430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.946598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4628.961305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4628.961468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4628.962181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.970286] Lustre: Skipped 1 previous similar message [ 4630.611623] Lustre: server umount lustre-OST0000 complete [ 4633.567943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4633.571885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.576119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.200142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.202187] Lustre: Skipped 3 previous similar messages [ 4644.320045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4644.322338] Lustre: Skipped 1 previous similar message [ 4646.367136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4646.424799] Lustre: server umount lustre-MDT0000 complete [ 4647.683291] LustreError: 209010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609787 with bad export cookie 7969668533998688316 [ 4647.686716] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4647.689192] LustreError: 209010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4647.815243] Lustre: server umount lustre-MDT0001 complete [ 4651.412079] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4652.483853] Key type lgssc unregistered [ 4652.609388] LNet: 212732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4652.613369] LNetError: 212732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4652.622328] LNet: Removed LNI 192.168.206.138@tcp [ 4652.924573] Key type .llcrypt unregistered [ 4652.926055] Key type ._llcrypt unregistered [ 4660.572595] Key type ._llcrypt registered [ 4660.573868] Key type .llcrypt registered [ 4660.614404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4660.949098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4660.957794] alg: No test for adler32 (adler32-zlib) [ 4661.821394] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4661.904871] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4663.487110] Key type lgssc registered [ 4663.848735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4666.458264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4666.465735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.625192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.968826] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4671.834959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.958419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4673.309990] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4675.611684] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4677.295300] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4679.444950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4679.570056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.485079] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4683.800696] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4692.900879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4698.080840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4698.081890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.088181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4698.090262] Lustre: Skipped 1 previous similar message [ 4699.603671] Lustre: server umount lustre-OST0000 complete [ 4702.687968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4702.691727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.697008] Lustre: Skipped 1 previous similar message [ 4702.699602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.094518] Lustre: server umount lustre-MDT0000 complete [ 4708.320440] LustreError: 216277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.328461] LustreError: 216277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4708.340584] LustreError: 213956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609848 with bad export cookie 17770016862532937434 [ 4708.344567] LustreError: 213956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4708.346409] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.472405] Lustre: server umount lustre-MDT0001 complete [ 4712.171088] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4713.167166] Key type lgssc unregistered [ 4713.294374] LNet: 217677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4713.298282] LNetError: 217677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4713.305468] LNet: Removed LNI 192.168.206.138@tcp [ 4713.573150] Key type .llcrypt unregistered [ 4713.574381] Key type ._llcrypt unregistered [ 4721.035877] Key type ._llcrypt registered [ 4721.036925] Key type .llcrypt registered [ 4721.072541] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4721.420827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4721.452966] alg: No test for adler32 (adler32-zlib) [ 4722.299751] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4722.381525] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4723.959154] Key type lgssc registered [ 4724.311021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4726.809431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4726.819076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.981740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.302123] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4731.948208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.075736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4733.373256] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4735.629917] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4737.309567] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4739.514805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.640847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.568702] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4743.955814] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4752.932960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4756.791195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4756.818753] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4756.901511] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4758.801589] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4761.135062] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4761.673728] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4772.034072] Lustre: Permanently deactivating lustre-OST0001 [ 4772.040053] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4773.499502] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4782.813615] Lustre: Permanently reactivating lustre-OST0001 [ 4782.818994] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4782.821528] Lustre: Skipped 2 previous similar messages [ 4782.974981] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.979367] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4782.982988] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4782.986495] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4782.998183] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4793.312352] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4793.312380] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.314632] LustreError: Skipped 1 previous similar message [ 4793.318183] Lustre: Skipped 1 previous similar message [ 4793.319870] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4793.322282] Lustre: Skipped 1 previous similar message [ 4796.899952] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4796.902494] Lustre: Skipped 1 previous similar message [ 4798.075466] Lustre: server umount lustre-OST0001 complete [ 4802.017783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.018895] LustreError: 220739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.022658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.027709] LustreError: 220739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4802.029575] Lustre: Skipped 1 previous similar message [ 4805.390672] Lustre: server umount lustre-OST0000 complete [ 4807.136690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.140775] Lustre: Skipped 1 previous similar message [ 4807.142562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4807.144614] Lustre: Skipped 1 previous similar message [ 4812.256201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4812.258011] Lustre: Skipped 1 previous similar message [ 4812.756333] Lustre: server umount lustre-MDT0000 complete [ 4814.006357] LustreError: 218901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609953 with bad export cookie 2270840427236143978 [ 4814.009258] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4814.011851] LustreError: 218901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4814.142541] Lustre: server umount lustre-MDT0001 complete [ 4817.684307] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4818.743345] Key type lgssc unregistered [ 4818.872274] LNet: 223961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4818.874594] LNetError: 223961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4818.880337] LNet: Removed LNI 192.168.206.138@tcp [ 4819.172678] Key type .llcrypt unregistered [ 4819.173781] Key type ._llcrypt unregistered [ 4832.582904] Key type ._llcrypt registered [ 4832.584658] Key type .llcrypt registered [ 4832.622451] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4832.919884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4832.930559] alg: No test for adler32 (adler32-zlib) [ 4833.785769] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4833.863429] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4835.439106] Key type lgssc registered [ 4835.797192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4838.337157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4838.344500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.436396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4839.443642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4839.501905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4840.754559] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4843.424686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.456101] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4843.466982] Lustre: 226142:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4843.562302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4844.871631] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4847.206835] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4849.052259] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4851.246515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4851.317946] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4851.396950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.373883] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4855.755873] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4856.869883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4862.806231] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4866.333600] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4873.892679] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4881.445642] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4891.032116] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4908.158948] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4908.161844] Lustre: Skipped 1 previous similar message [ 4947.968705] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4947.972763] Lustre: Skipped 4 previous similar messages [ 5014.206194] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5014.209363] Lustre: Skipped 8 previous similar messages [ 5023.713953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.714934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5023.718405] Lustre: Skipped 1 previous similar message [ 5023.722115] Lustre: Skipped 1 previous similar message [ 5028.834218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5028.836956] Lustre: Skipped 1 previous similar message [ 5028.895550] Lustre: server umount lustre-OST0000 complete [ 5032.927929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5032.931399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.937519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.953343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.311543] Lustre: server umount lustre-MDT0000 complete [ 5037.582480] LustreError: 228762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610177 with bad export cookie 9933765782814737907 [ 5037.586080] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5037.587114] LustreError: 228762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5037.710595] Lustre: server umount lustre-MDT0001 complete [ 5041.535625] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5042.610256] Key type lgssc unregistered [ 5042.752354] LNet: 230919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5042.756227] LNetError: 230919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5042.766436] LNet: Removed LNI 192.168.206.138@tcp [ 5043.048983] Key type .llcrypt unregistered [ 5043.050238] Key type ._llcrypt unregistered [ 5052.131662] Key type ._llcrypt registered [ 5052.133239] Key type .llcrypt registered [ 5052.171953] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5052.619075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5052.644422] alg: No test for adler32 (adler32-zlib) [ 5053.501043] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5053.579591] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5055.159125] Key type lgssc registered [ 5055.540922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5058.226769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5058.234505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.419552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.847147] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5063.719678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.852815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5065.166456] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5067.543280] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5069.272160] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5071.640957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.783788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.737425] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5076.152969] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5086.118511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5091.296716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5091.298323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.303676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.305594] Lustre: Skipped 1 previous similar message [ 5096.417914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5096.420016] Lustre: Skipped 1 previous similar message [ 5097.287304] Lustre: server umount lustre-OST0000 complete [ 5099.999918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5100.002623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.006590] Lustre: Skipped 1 previous similar message [ 5100.008435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5101.536960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.601205] Lustre: server umount lustre-MDT0000 complete [ 5105.779307] LustreError: 232290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610245 with bad export cookie 17087103826456856325 [ 5105.783339] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.784430] LustreError: 232290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5105.903128] Lustre: server umount lustre-MDT0001 complete [ 5109.332191] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5110.343416] Key type lgssc unregistered [ 5110.473542] LNet: 236011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5110.476322] LNetError: 236011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5110.487437] LNet: Removed LNI 192.168.206.138@tcp [ 5110.824173] Key type .llcrypt unregistered [ 5110.825223] Key type ._llcrypt unregistered [ 5119.629614] Key type ._llcrypt registered [ 5119.630727] Key type .llcrypt registered [ 5119.667092] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5120.046977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5120.092847] alg: No test for adler32 (adler32-zlib) [ 5120.939075] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5121.013667] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5122.591113] Key type lgssc registered [ 5122.922816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5387.160306] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5388.095208] Key type lgssc unregistered [ 5388.209336] LNet: 238008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5388.212416] LNetError: 238008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5388.223274] LNet: Removed LNI 192.168.206.138@tcp [ 5388.499355] Key type .llcrypt unregistered [ 5388.500814] Key type ._llcrypt unregistered [ 5396.297427] Key type ._llcrypt registered [ 5396.298523] Key type .llcrypt registered [ 5396.339678] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5403.186143] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5403.519091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5403.577952] alg: No test for adler32 (adler32-zlib) [ 5404.424655] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5404.500791] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5406.079092] Key type lgssc registered [ 5406.426280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5416.295677] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5020 sec ========== 07:49:16 (1781610556) [ 5416.888889] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:49:17 (1781610557) === [ 5418.129263] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:49:18 (1781610558) === [ 5430.884460] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5431.883532] Key type lgssc unregistered [ 5432.001341] LNet: 243213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5432.004663] LNetError: 243213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5432.013301] LNet: Removed LNI 192.168.206.138@tcp [ 5432.289516] Key type .llcrypt unregistered [ 5432.290827] Key type ._llcrypt unregistered