[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 662032271 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003368] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008526] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010005] pid_max: default: 32768 minimum: 301 [ 0.011039] LSM: Security Framework initializing [ 0.011887] Yama: becoming mindful. [ 0.012019] SELinux: Initializing. [ 0.013035] *** VALIDATE selinux *** [ 0.021021] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026262] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] *** VALIDATE tmpfs *** [ 0.031359] *** VALIDATE proc *** [ 0.032207] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.035204] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038017] Spectre V2 : User space: Vulnerable [ 0.039003] Speculative Store Bypass: Vulnerable [ 0.041838] debug: unmapping init [mem 0xffffffffae259000-0xffffffffae260fff] [ 0.043209] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049008] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.051006] ... event mask: 000000070000000f [ 0.052262] rcu: Hierarchical SRCU implementation. [ 0.054641] smp: Bringing up secondary CPUs ... [ 0.055570] x86: Booting SMP configuration: [ 0.056012] .... node #0, CPUs: #1 #2 #3 [ 0.060018] smp: Brought up 1 node, 4 CPUs [ 0.062008] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.100277] node 0 deferred pages initialised in 36ms [ 0.106403] devtmpfs: initialized [ 0.109010] x86/mm: Memory block size: 128MB [ 0.119494] gcov: version magic: 0x41383552 [ 0.142341] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145232] pinctrl core: initialized pinctrl subsystem [ 0.147159] [ 0.147853] ************************************************************* [ 0.151010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153008] ** ** [ 0.156009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158007] ** ** [ 0.160007] ** This means that this kernel is built to expose internal ** [ 0.163010] ** IOMMU data structures, which may compromise security on ** [ 0.166010] ** your system. ** [ 0.169011] ** ** [ 0.172009] ** If you see this message and you are not debugging the ** [ 0.174014] ** kernel, report this immediately to your vendor! ** [ 0.177011] ** ** [ 0.180008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182011] ************************************************************* [ 0.185952] NET: Registered protocol family 16 [ 0.187561] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191199] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207095] cpuidle: using governor menu [ 0.209958] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213504] PCI: Using configuration type 1 for base access [ 0.216187] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226169] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230240] cryptd: max_cpu_qlen set to 1000 [ 0.231505] ACPI: Added _OSI(Module Device) [ 0.232000] ACPI: Added _OSI(Processor Device) [ 0.233008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241302] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248329] ACPI: Interpreter enabled [ 0.250044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251008] ACPI: Using IOAPIC for interrupt routing [ 0.254089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269740] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284681] acpiphp: Slot [2] registered [ 0.287176] acpiphp: Slot [5] registered [ 0.288085] acpiphp: Slot [6] registered [ 0.290104] acpiphp: Slot [7] registered [ 0.291071] acpiphp: Slot [8] registered [ 0.293139] acpiphp: Slot [9] registered [ 0.294093] acpiphp: Slot [10] registered [ 0.296200] acpiphp: Slot [3] registered [ 0.297062] acpiphp: Slot [4] registered [ 0.299061] acpiphp: Slot [11] registered [ 0.300079] acpiphp: Slot [12] registered [ 0.302071] acpiphp: Slot [13] registered [ 0.304095] acpiphp: Slot [14] registered [ 0.305073] acpiphp: Slot [15] registered [ 0.307070] acpiphp: Slot [16] registered [ 0.308087] acpiphp: Slot [17] registered [ 0.310070] acpiphp: Slot [18] registered [ 0.311104] acpiphp: Slot [19] registered [ 0.313070] acpiphp: Slot [20] registered [ 0.315066] acpiphp: Slot [21] registered [ 0.316064] acpiphp: Slot [22] registered [ 0.318096] acpiphp: Slot [23] registered [ 0.319230] acpiphp: Slot [24] registered [ 0.321062] acpiphp: Slot [25] registered [ 0.322064] acpiphp: Slot [26] registered [ 0.324079] acpiphp: Slot [27] registered [ 0.325115] acpiphp: Slot [28] registered [ 0.327079] acpiphp: Slot [29] registered [ 0.329080] acpiphp: Slot [30] registered [ 0.330062] acpiphp: Slot [31] registered [ 0.332066] PCI host bridge to bus 0000:00 [ 0.333012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348146] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352076] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355266] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365714] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371035] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381725] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384841] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391009] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425080] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.442032] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543356] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.578020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.626019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.648168] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.666027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.676019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.693010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.702614] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.714015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.722026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.759016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.773000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.784015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.794015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.810019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.824875] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.833014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.841013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.863018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.877577] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.880326] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.882403] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.884334] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.886158] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.890716] iommu: Default domain type: Passthrough [ 0.892427] SCSI subsystem initialized [ 0.894129] ACPI: bus type USB registered [ 0.896105] usbcore: registered new interface driver usbfs [ 0.898085] usbcore: registered new interface driver hub [ 0.900102] usbcore: registered new device driver usb [ 0.904160] pps_core: LinuxPPS API ver. 1 registered [ 0.906011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.911058] PTP clock support registered [ 0.916049] EDAC MC: Ver: 3.0.0 [ 0.919138] PCI: Using ACPI for IRQ routing [ 0.927012] NetLabel: Initializing [ 0.928059] NetLabel: domain hash size = 128 [ 0.930007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.932092] NetLabel: unlabeled traffic allowed by default [ 0.934106] vgaarb: loaded [ 0.936232] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.937010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.941860] clocksource: Switched to clocksource kvm-clock [ 1.062269] VFS: Disk quotas dquot_6.6.0 [ 1.063640] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.066874] *** VALIDATE ramfs *** [ 1.068465] *** VALIDATE hugetlbfs *** [ 1.070074] pnp: PnP ACPI init [ 1.072397] pnp: PnP ACPI: found 6 devices [ 1.089121] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.092228] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.094366] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.096765] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.099573] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.103466] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.106755] NET: Registered protocol family 2 [ 1.109940] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.114890] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.118736] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.128356] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.132397] TCP: Hash tables configured (established 65536 bind 65536) [ 1.135647] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.139404] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.143790] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.146869] NET: Registered protocol family 1 [ 1.150087] RPC: Registered named UNIX socket transport module. [ 1.152192] RPC: Registered udp transport module. [ 1.154966] RPC: Registered tcp transport module. [ 1.156686] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.158995] NET: Registered protocol family 44 [ 1.161037] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.163958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.166022] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.168473] PCI: CLS 0 bytes, default 64 [ 1.170207] Unpacking initramfs... [ 3.383951] debug: unmapping init [mem 0xffff90fffcc54000-0xffff90fffffbffff] [ 3.389562] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.392318] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.397793] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.146548] Initialise system trusted keyrings [ 4.148453] Key type blacklist registered [ 4.150452] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.164273] zbud: loaded [ 4.169235] *** VALIDATE nfs *** [ 4.170489] *** VALIDATE nfs4 *** [ 4.174957] pstore: using deflate compression [ 4.179747] Platform Keyring initialized [ 4.367225] NET: Registered protocol family 38 [ 4.369987] Key type asymmetric registered [ 4.373677] Asymmetric key parser 'x509' registered [ 4.376443] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.381454] io scheduler mq-deadline registered [ 4.384873] io scheduler kyber registered [ 4.389818] io scheduler bfq registered [ 4.391860] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.396689] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.402281] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.410371] ACPI: Power Button [PWRF] [ 4.420956] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.431993] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.456466] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.473367] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.503635] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.534471] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.570115] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.575535] Non-volatile memory driver v1.3 [ 4.578459] Linux agpgart interface v0.103 [ 4.631217] virtio_blk virtio1: [vda] 145784 512-byte logical blocks (74.6 MB/71.2 MiB) [ 4.634393] vda: detected capacity change from 0 to 74641408 [ 4.667391] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.671210] vdb: detected capacity change from 0 to 1073741824 [ 4.693670] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.696784] vdc: detected capacity change from 0 to 2621440000 [ 4.752189] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.757317] vdd: detected capacity change from 0 to 2621440000 [ 4.774603] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.778584] vde: detected capacity change from 0 to 4294967296 [ 4.801019] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.805243] vdf: detected capacity change from 0 to 4294967296 [ 4.811436] libphy: Fixed MDIO Bus: probed [ 4.818566] usbcore: registered new interface driver usbserial_generic [ 4.822236] usbserial: USB Serial support registered for generic [ 4.824855] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.830735] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.834288] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.837995] mousedev: PS/2 mouse device common for all mice [ 4.842780] rtc_cmos 00:05: RTC can wake from S4 [ 4.846076] rtc_cmos 00:05: registered as rtc0 [ 4.848323] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.853386] intel_pstate: CPU model not supported [ 4.858025] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.863478] hid: raw HID events driver (C) Jiri Kosina [ 4.865408] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.868046] usbcore: registered new interface driver usbhid [ 4.874706] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.877470] usbhid: USB HID core driver [ 4.877626] drop_monitor: Initializing network drop monitor service [ 4.877741] Initializing XFRM netlink socket [ 4.878578] NET: Registered protocol family 10 [ 4.894956] Segment Routing with IPv6 [ 4.897432] NET: Registered protocol family 17 [ 4.899377] mpls_gso: MPLS GSO support [ 4.905925] RAS: Correctable Errors collector initialized. [ 4.908464] AVX version of gcm_enc/dec engaged. [ 4.910377] AES CTR mode by8 optimization enabled [ 5.023915] sched_clock: Marking stable (5023790158, 0)->(6066142345, -1042352187) [ 5.027872] registered taskstats version 1 [ 5.032322] Loading compiled-in X.509 certificates [ 5.034916] zswap: loaded using pool lzo/zbud [ 5.068994] Key type big_key registered [ 5.088950] Key type encrypted registered [ 5.091068] ima: No TPM chip found, activating TPM-bypass! [ 5.093696] ima: Allocated hash algorithm: sha1 [ 5.095680] ima: No architecture policies found [ 5.097786] evm: Initialising EVM extended attributes: [ 5.100183] evm: security.selinux [ 5.101638] evm: security.ima [ 5.102933] evm: security.capability [ 5.104649] evm: HMAC attrs: 0x1 [ 5.107745] rtc_cmos 00:05: setting system clock to 2026-07-15 13:25:28 UTC (1784121928) [ 5.116982] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 5.121160] debug: unmapping init [mem 0xffffffffadf82000-0xffffffffae258fff] [ 5.129213] Write protecting the kernel read-only data: 28672k [ 5.132926] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 5.135694] debug: unmapping init [mem 0xffffffffacf14000-0xffffffffacffffff] [ 5.182397] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.191987] systemd[1]: Detected virtualization kvm. [ 5.193924] systemd[1]: Detected architecture x86-64. [ 5.196523] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.251798] systemd[1]: No hostname configured. [ 5.254318] systemd[1]: Set hostname to . [ 5.257939] random: systemd: uninitialized urandom read (16 bytes read) [ 5.262530] systemd[1]: Initializing machine ID from random generator. [ 5.423448] random: ln: uninitialized urandom read (6 bytes read) [ 5.569755] random: systemd: uninitialized urandom read (16 bytes read) [ 5.572881] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.585070] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.593339] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.378975] device-mapper: uevent: version 1.0.3 [ 7.385815] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook..[ 9.297664] virtio_net virtio0 ens2: renamed from eth0 . [ 9.428843] random: fast init done [ 9.941530] scsi host0: ata_piix [ 10.171943] scsi host1: ata_piix [ 10.184549] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.195246] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.618400] random: crng init done [ 14.626322] random: 7 urandom warning(s) missed due to ratelimiting [ 17.915669] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.979229] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.972984] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.535130] SELinux: Disabled at runtime. [ 21.629262] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.639369] systemd[1]: Detected virtualization kvm. [ 21.641796] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.254739] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.258764] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.266114] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.275644] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.279753] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.290406] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.298909] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ 23.676086] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 24.960757] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.845797] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.949738] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.341614] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.466501] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 31.246574] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** [ 32.171880] NFS: Registering the id_resolver key type [ 32.174321] Key type id_resolver registered ] A start job is running for Configur…-only root support (8s / no limit)[ 32.176869] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-server login: [ 62.892819] libcfs: loading out-of-tree module taints kernel. [ 62.914837] Key type ._llcrypt registered [ 62.916686] Key type .llcrypt registered [ 62.972816] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 71.119354] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 71.826986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.834142] alg: No test for adler32 (adler32-zlib) [ 72.817486] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 73.117584] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 74.735151] Key type lgssc registered [ 75.274472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.180535] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.854610] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 102.825777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 102.834188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 103.932911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 103.951595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 103.991102] Lustre: lustre-MDT0000: new disk, initializing [ 104.037090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.048953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 105.840647] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.741630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.786029] Lustre: 6486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.814314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.818581] Lustre: Skipped 1 previous similar message [ 111.872854] Lustre: lustre-MDT0001: new disk, initializing [ 111.905559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.920396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 111.930211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 113.547236] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.016528] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 119.445100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 119.538863] Lustre: lustre-OST0000: new disk, initializing [ 119.540899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 119.544581] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 119.569133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.521809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 121.526172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 121.552857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 121.963548] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.882570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.932089] Lustre: lustre-OST0001: new disk, initializing [ 128.933847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 128.936401] Lustre: 9498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 128.963584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.605647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 130.610437] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 130.640893] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 131.424114] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.827625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 142.694680] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 144.979555] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing check_logdir /tmp/testlogs/ [ 146.629499] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing yml_node [ 148.321855] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 149.331914] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 150.250456] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 150.830168] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 15 09:27:53 EDT 2026 [ 156.821300] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 157.287455] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 171.488373] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 171.489151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 171.493898] Lustre: Skipped 3 previous similar messages [ 171.496186] Lustre: Skipped 2 previous similar messages [ 175.193519] Lustre: server umount lustre-MDT0000 complete [ 176.445783] LustreError: 7434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122099 with bad export cookie 9382508117624582238 [ 176.451430] LustreError: 7434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 176.453268] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 176.607621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.607820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.611895] Lustre: Skipped 1 previous similar message [ 176.614268] Lustre: Skipped 1 previous similar message [ 181.728394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.728452] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 181.732049] Lustre: Skipped 2 previous similar messages [ 181.739799] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 182.641823] Lustre: server umount lustre-MDT0001 complete [ 190.413599] Lustre: server umount lustre-OST0000 complete [ 198.222046] Lustre: server umount lustre-OST0001 complete [ 200.637176] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 203.927487] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 218.870486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.966157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.988780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.029037] Lustre: lustre-MDT0000: new disk, initializing [ 219.061360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.069164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.818964] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 225.808043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.870290] Lustre: 15037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.023547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.853229] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 232.466349] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 234.809098] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 237.685036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.809850] Lustre: lustre-OST0000: new disk, initializing [ 237.812369] Lustre: Skipped 1 previous similar message [ 237.814604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.817264] Lustre: Skipped 2 previous similar messages [ 237.821069] Lustre: 17007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.858398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.860660] Lustre: Skipped 1 previous similar message [ 239.048776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.052921] Lustre: Skipped 1 previous similar message [ 239.057539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.097915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.297634] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 244.716972] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 246.585586] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 246.679307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 248.387529] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 248.473181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 249.315409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 249.315589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.318194] LustreError: Skipped 1 previous similar message [ 249.325867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 249.330447] Lustre: Skipped 1 previous similar message [ 254.438838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 254.444346] Lustre: Skipped 1 previous similar message [ 255.408446] Lustre: server umount lustre-OST0000 complete [ 259.552843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.558488] Lustre: Skipped 2 previous similar messages [ 263.280613] Lustre: server umount lustre-MDT0000 complete [ 264.672236] LustreError: 15849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.678225] LustreError: 15849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 264.817485] LustreError: 15030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122188 with bad export cookie 9382508117624583771 [ 264.818811] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 264.824118] LustreError: 15030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 264.954649] Lustre: server umount lustre-MDT0001 complete [ 266.248647] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:29:48 (1784122188) [ 269.298715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.510836] LustreError: 19266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.554069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 271.198536] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 274.912410] LustreError: 19267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.029941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.465242] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 281.115767] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 283.995463] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 287.561645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 290.722372] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 294.673498] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 309.221553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 309.227518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.238190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 309.242948] Lustre: Skipped 2 previous similar messages [ 310.244343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 314.407069] Lustre: server umount lustre-OST0000 complete [ 320.481758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.491576] Lustre: Skipped 2 previous similar messages [ 322.783779] Lustre: server umount lustre-MDT0000 complete [ 324.939798] LustreError: 19246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122248 with bad export cookie 9382508117624584653 [ 324.942518] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.948736] LustreError: 19246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 325.183195] Lustre: server umount lustre-MDT0001 complete [ 330.787944] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 332.233279] Key type lgssc unregistered [ 332.424091] LNet: 22814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.429978] LNetError: 22814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.485735] LNet: Removed LNI 192.168.202.135@tcp [ 334.177717] Key type .llcrypt unregistered [ 334.180793] Key type ._llcrypt unregistered [ 338.098011] hrtimer: interrupt took 3482451 ns [ 347.657552] Key type ._llcrypt registered [ 347.659896] Key type .llcrypt registered [ 347.737169] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 348.377626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 348.388773] alg: No test for adler32 (adler32-zlib) [ 349.272343] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 349.386267] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 351.007153] Key type lgssc registered [ 351.634201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 356.638772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 356.660433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.939882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.684086] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 366.130634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.326734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.565864] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 374.288773] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 377.376330] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 380.377054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 380.531590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.913452] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 383.594913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 386.139693] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 396.258320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.263339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 396.265990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 396.267027] Lustre: Skipped 1 previous similar message [ 396.280192] LustreError: Skipped 1 previous similar message [ 397.263337] Lustre: server umount lustre-OST0000 complete [ 401.379200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.384073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.384766] Lustre: Skipped 1 previous similar message [ 401.391456] Lustre: Skipped 2 previous similar messages [ 405.751606] Lustre: server umount lustre-MDT0000 complete [ 406.496366] LustreError: 26048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.503386] LustreError: 26048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 407.495132] LustreError: 24189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122330 with bad export cookie 9177945184015917586 [ 407.496805] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 407.501139] LustreError: 24189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 407.514408] LustreError: 24938:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 407.521599] LustreError: 24938:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 407.676913] Lustre: server umount lustre-MDT0001 complete [ 412.703791] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 413.944142] Key type lgssc unregistered [ 414.086434] LNet: 28177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.089566] LNetError: 28177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 414.098343] LNet: Removed LNI 192.168.202.135@tcp [ 414.480927] Key type .llcrypt unregistered [ 414.483099] Key type ._llcrypt unregistered [ 425.828829] Key type ._llcrypt registered [ 425.830468] Key type .llcrypt registered [ 425.879891] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 426.435928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 426.448242] alg: No test for adler32 (adler32-zlib) [ 427.355433] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 427.500463] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 429.111746] Key type lgssc registered [ 429.707158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 433.818486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 433.834347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.069139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.242256] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 441.622411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.836121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.589501] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 447.067989] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 449.694688] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.634790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.819769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.715107] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 459.947271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 461.923418] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.552889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.557041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.562910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.364031] Lustre: server umount lustre-OST0000 complete [ 477.665758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.666669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 477.677084] Lustre: Skipped 1 previous similar message [ 477.677875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.688252] Lustre: Skipped 2 previous similar messages [ 481.247217] Lustre: server umount lustre-MDT0000 complete [ 482.784824] LustreError: 30147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.792172] LustreError: 30147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 482.890176] LustreError: 29548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122406 with bad export cookie 4617347474022147476 [ 482.892811] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.895311] LustreError: 29548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 483.048704] Lustre: server umount lustre-MDT0001 complete [ 487.889959] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 489.472880] Key type lgssc unregistered [ 489.643496] LNet: 33437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.648398] LNetError: 33437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.659416] LNet: Removed LNI 192.168.202.135@tcp [ 490.101746] Key type .llcrypt unregistered [ 490.103739] Key type ._llcrypt unregistered [ 502.255038] Key type ._llcrypt registered [ 502.257632] Key type .llcrypt registered [ 502.334478] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 502.699420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 502.755112] alg: No test for adler32 (adler32-zlib) [ 503.665507] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 503.787268] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 505.375330] Key type lgssc registered [ 506.028579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 510.504138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 510.519420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.792549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.188675] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 519.280195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.512870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 521.717308] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 525.600163] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.563352] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 531.768127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 531.953658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.787893] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 537.064699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 538.172709] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.297860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 547.301314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.307121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.347275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.349447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.356860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.851245] Lustre: server umount lustre-OST0000 complete [ 555.491979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.495466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.501370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 555.505935] Lustre: Skipped 1 previous similar message [ 559.586159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.586318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.592635] Lustre: Skipped 1 previous similar message [ 564.704399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.707917] Lustre: Skipped 1 previous similar message [ 569.311220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 569.419942] Lustre: server umount lustre-MDT0000 complete [ 571.737450] LustreError: 34810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122495 with bad export cookie 13921426120665722110 [ 571.738972] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.749609] LustreError: 34810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 571.983424] Lustre: server umount lustre-MDT0001 complete [ 577.914121] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 579.745657] Key type lgssc unregistered [ 579.969618] LNet: 38357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.975971] LNetError: 38357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.987638] LNet: Removed LNI 192.168.202.135@tcp [ 580.565652] Key type .llcrypt unregistered [ 580.567275] Key type ._llcrypt unregistered [ 592.390129] Key type ._llcrypt registered [ 592.391738] Key type .llcrypt registered [ 592.434988] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 592.914506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 592.926508] alg: No test for adler32 (adler32-zlib) [ 593.804724] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 593.907838] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 595.503258] Key type lgssc registered [ 596.040685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.726591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 599.735700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.931787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.733180] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 606.536587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.736288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 608.433278] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 611.281322] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.344205] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.157982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 616.310950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.630196] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 621.583769] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 626.533686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 631.776906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.779735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 631.790124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 631.795634] Lustre: Skipped 1 previous similar message [ 632.334773] Lustre: lustre-OST0000: Not available for connect from 192.168.202.35@tcp (stopping) [ 635.736267] Lustre: server umount lustre-OST0000 complete [ 649.228478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.229439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.234079] Lustre: Skipped 2 previous similar messages [ 652.321176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.323565] Lustre: Skipped 2 previous similar messages [ 655.399166] Lustre: server umount lustre-MDT0000 complete [ 656.732782] LustreError: 39726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122580 with bad export cookie 17063853252222138633 [ 656.735338] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.737492] LustreError: 39726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 671.199103] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 671.302687] Lustre: server umount lustre-MDT0001 complete [ 675.464826] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 676.595562] Key type lgssc unregistered [ 676.739586] LNet: 43453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.743256] LNetError: 43453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.753614] LNet: Removed LNI 192.168.202.135@tcp [ 677.105519] Key type .llcrypt unregistered [ 677.106941] Key type ._llcrypt unregistered [ 686.459161] Key type ._llcrypt registered [ 686.460879] Key type .llcrypt registered [ 686.504320] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 686.849092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.888201] alg: No test for adler32 (adler32-zlib) [ 687.768849] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 687.875544] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 689.471143] Key type lgssc registered [ 689.890344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.778282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 692.786655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.957476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.485461] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 698.541927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.724228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 700.339946] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 703.435593] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 705.806710] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 708.290631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.459040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.022836] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 714.448218] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 718.701204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 723.937779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.942252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 723.948197] Lustre: Skipped 1 previous similar message [ 725.006656] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [ 725.009808] Lustre: Skipped 2 previous similar messages [ 728.411836] Lustre: server umount lustre-MDT0000 complete [ 729.056022] LustreError: 44843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.063471] LustreError: 44843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 729.833813] LustreError: 44869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122653 with bad export cookie 5801200561054367041 [ 729.837136] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.840380] LustreError: 44869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 730.116889] LustreError: 46138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.128138] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [ 734.178471] LustreError: 46138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.179201] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.192267] Lustre: Skipped 1 previous similar message [ 734.194546] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 736.046607] Lustre: server umount lustre-MDT0001 complete [ 762.335134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 762.404322] Lustre: server umount lustre-OST0000 complete [ 768.896484] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 770.055089] Key type lgssc unregistered [ 770.203331] LNet: 48701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.207653] LNetError: 48701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.220600] LNet: Removed LNI 192.168.202.135@tcp [ 770.584188] Key type .llcrypt unregistered [ 770.585923] Key type ._llcrypt unregistered [ 781.426326] Key type ._llcrypt registered [ 781.427753] Key type .llcrypt registered [ 781.466817] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 781.807734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 781.839092] alg: No test for adler32 (adler32-zlib) [ 782.729262] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 782.838986] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 784.439162] Key type lgssc registered [ 784.945312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 788.222509] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 788.232984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.679403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 808.799439] LustreError: 50065:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 835.423717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.741243] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 840.644255] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1003.490535] Lustre: server umount lustre-OST0000 complete [ 1009.055224] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1010.091120] Key type lgssc unregistered [ 1010.219408] LNet: 51741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.222863] LNetError: 51741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.230321] LNet: Removed LNI 192.168.202.135@tcp [ 1010.548611] Key type .llcrypt unregistered [ 1010.549753] Key type ._llcrypt unregistered [ 1019.151944] Key type ._llcrypt registered [ 1019.153686] Key type .llcrypt registered [ 1019.192404] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1019.507886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.516680] alg: No test for adler32 (adler32-zlib) [ 1020.364296] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1020.449962] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1022.023153] Key type lgssc registered [ 1022.411867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.264196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.279116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.504793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.181976] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1031.344876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.494152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.031890] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1035.557543] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1037.393849] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1039.740610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.859779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.864584] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1044.394360] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.985420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1045.986165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.996633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.255808] Lustre: server umount lustre-OST0000 complete [ 1048.512100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.512744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.516389] Lustre: Skipped 1 previous similar message [ 1053.664126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.666504] Lustre: Skipped 2 previous similar messages [ 1054.683081] Lustre: server umount lustre-MDT0000 complete [ 1055.994491] LustreError: 53112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784122979 with bad export cookie 7460263760603866984 [ 1055.998236] LustreError: 53112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1055.998411] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.105158] Lustre: server umount lustre-MDT0001 complete [ 1059.507414] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1060.533081] Key type lgssc unregistered [ 1060.663373] LNet: 56655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.666370] LNetError: 56655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.674334] LNet: Removed LNI 192.168.202.135@tcp [ 1060.985395] Key type .llcrypt unregistered [ 1060.987175] Key type ._llcrypt unregistered [ 1069.821554] Key type ._llcrypt registered [ 1069.822838] Key type .llcrypt registered [ 1069.862811] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1070.159652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.184413] alg: No test for adler32 (adler32-zlib) [ 1071.034408] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1071.120338] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1072.703143] Key type lgssc registered [ 1073.057643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1075.588569] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1075.594749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1091.039389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1096.159282] LustreError: 58018:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 1122.783243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.701836] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1127.120579] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1129.408173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.595034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.621137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1130.899957] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1133.908098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.049488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.477094] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1138.015877] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1139.832151] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.241235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.243086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.246656] Lustre: Skipped 1 previous similar message [ 1146.635530] Lustre: server umount lustre-OST0000 complete [ 1160.678994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.679950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1160.685040] Lustre: Skipped 1 previous similar message [ 1160.687092] Lustre: Skipped 1 previous similar message [ 1161.696276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.698179] Lustre: Skipped 2 previous similar messages [ 1166.816020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.818226] Lustre: Skipped 1 previous similar message [ 1166.873133] Lustre: server umount lustre-MDT0000 complete [ 1168.134120] LustreError: 58026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123091 with bad export cookie 4635845243139450829 [ 1168.136801] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.138554] LustreError: 58026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1174.313997] Lustre: server umount lustre-MDT0001 complete [ 1178.010366] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1179.053911] Key type lgssc unregistered [ 1179.186331] LNet: 61755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.188694] LNetError: 61755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.198479] LNet: Removed LNI 192.168.202.135@tcp [ 1179.504087] Key type .llcrypt unregistered [ 1179.505860] Key type ._llcrypt unregistered [ 1188.746275] Key type ._llcrypt registered [ 1188.747644] Key type .llcrypt registered [ 1188.788500] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1189.186032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.193930] alg: No test for adler32 (adler32-zlib) [ 1190.049393] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1190.143715] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1191.735199] Key type lgssc registered [ 1192.114265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.816225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.821696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.962289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.399271] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1200.521764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.642703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.014633] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1204.359218] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1206.104940] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1208.277975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.388633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.400192] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1213.212977] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1221.733775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1227.233231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1227.234519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.240710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.243853] Lustre: Skipped 1 previous similar message [ 1227.858902] Lustre: server umount lustre-OST0000 complete [ 1231.328226] Lustre: server umount lustre-MDT0000 complete [ 1231.328407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1231.333802] LustreError: Skipped 1 previous similar message [ 1231.336766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.342334] Lustre: Skipped 1 previous similar message [ 1231.344997] LustreError: 63163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.349872] LustreError: 63163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1232.352111] LustreError: 63144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.352686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.754539] LustreError: 65642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123156 with bad export cookie 12189084185835252089 [ 1232.757929] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.759321] LustreError: 65642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1232.892275] Lustre: server umount lustre-MDT0001 complete [ 1236.900058] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1238.078081] Key type lgssc unregistered [ 1238.211446] LNet: 66692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.214843] LNetError: 66692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.224322] LNet: Removed LNI 192.168.202.135@tcp [ 1238.555699] Key type .llcrypt unregistered [ 1238.556904] Key type ._llcrypt unregistered [ 1248.692690] Key type ._llcrypt registered [ 1248.693550] Key type .llcrypt registered [ 1248.737957] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1249.215924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.226750] alg: No test for adler32 (adler32-zlib) [ 1250.098224] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1250.201548] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1251.791201] Key type lgssc registered [ 1252.219895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.625418] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:46:18 (1784123178) [ 1258.072878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1258.079713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.236243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.739912] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1264.203210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.388215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1266.105248] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1269.069083] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1271.189461] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1273.843831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.963501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.073532] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1278.995201] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1284.197926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1291.750865] Lustre: Failing over lustre-MDT0000 [ 1291.941258] Lustre: server umount lustre-MDT0000 complete [ 1294.817440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.821929] Lustre: Skipped 1 previous similar message [ 1294.822475] LustreError: 68366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.830956] LustreError: 68366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1295.530074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.578633] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.636625] Lustre: *** cfs_fail_loc=135, val=0*** [ 1295.684914] LustreError: 71475:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1295.688197] LustreError: 71475:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.135@tcp: cfg command failed: rc = -2 [ 1295.691583] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1295.691583] [ 1295.695058] LustreError: MGC192.168.202.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1295.701907] LustreError: 71463:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1295.706331] LustreError: 71463:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1295.711456] LustreError: 71463:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1295.714740] LustreError: 71463:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1295.732344] Lustre: server umount lustre-MDT0000 complete [ 1295.734866] LustreError: 71463:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1295.885706] LustreError: 68366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.382234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.422881] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.517589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.552742] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1299.961351] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1300.997661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1303.529540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1303.539516] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1303.558153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1305.056105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.059241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.064896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.641891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.641896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.646611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.757558] Lustre: server umount lustre-OST0000 complete [ 1313.760704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.761275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.769173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.771175] Lustre: Skipped 2 previous similar messages [ 1318.242810] Lustre: server umount lustre-MDT0000 complete [ 1318.881312] LustreError: 68370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.887641] LustreError: 68370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1319.885659] LustreError: 70967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123243 with bad export cookie 5562206192765192225 [ 1319.890332] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.890520] LustreError: 70967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1320.028158] Lustre: server umount lustre-MDT0001 complete [ 1323.801244] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1324.864498] Key type lgssc unregistered [ 1325.004388] LNet: 73253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.007316] LNetError: 73253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.016344] LNet: Removed LNI 192.168.202.135@tcp [ 1325.325354] Key type .llcrypt unregistered [ 1325.326357] Key type ._llcrypt unregistered [ 1334.694271] Key type ._llcrypt registered [ 1334.695357] Key type .llcrypt registered [ 1334.735614] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1335.164433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.172580] alg: No test for adler32 (adler32-zlib) [ 1336.038202] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1336.142944] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1337.735234] Key type lgssc registered [ 1338.164135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1341.516526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1341.523313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.706711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.398583] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1347.441061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.582349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.064569] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1351.507938] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.433674] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1355.777860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.890134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.986218] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1360.475756] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1369.190527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1373.500902] Lustre: Failing over lustre-MDT0000 [ 1373.707441] Lustre: server umount lustre-MDT0000 complete [ 1374.688991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.690063] LustreError: 74643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.693243] Lustre: Skipped 2 previous similar messages [ 1374.699984] LustreError: 74643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1376.268842] LustreError: 74668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.875775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.918026] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.971993] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1377.022539] LustreError: 77749:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1377.025729] LustreError: 77749:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.135@tcp: cfg command failed: rc = -9 [ 1377.029543] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1377.029543] [ 1377.033448] LustreError: MGC192.168.202.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1377.040922] LustreError: 77737:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1377.045379] LustreError: 77737:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1377.050077] LustreError: 77737:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1377.053499] LustreError: 77737:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1377.065695] Lustre: server umount lustre-MDT0000 complete [ 1377.067213] LustreError: 77737:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1379.484029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.546806] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.645379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.667129] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1381.087400] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1381.380618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1384.933742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1384.941580] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1384.958678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1387.999932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1388.002194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.006610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.049805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.050447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.053496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.952410] Lustre: server umount lustre-OST0000 complete [ 1395.170622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1395.170667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.177994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1395.180495] Lustre: Skipped 2 previous similar messages [ 1399.249261] Lustre: server umount lustre-MDT0000 complete [ 1400.288030] LustreError: 74644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.293079] LustreError: 74644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1400.452923] LustreError: 74625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123323 with bad export cookie 12677610792397808551 [ 1400.455938] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1400.456261] LustreError: 74625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1400.549506] Lustre: server umount lustre-MDT0001 complete [ 1403.911960] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1404.943873] Key type lgssc unregistered [ 1405.075440] LNet: 79522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.078141] LNetError: 79522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.084464] LNet: Removed LNI 192.168.202.135@tcp [ 1405.392871] Key type .llcrypt unregistered [ 1405.394522] Key type ._llcrypt unregistered [ 1409.470154] Key type ._llcrypt registered [ 1409.471446] Key type .llcrypt registered [ 1409.512489] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1409.631696] Key type .llcrypt unregistered [ 1409.633317] Key type ._llcrypt unregistered [ 1418.979769] Key type ._llcrypt registered [ 1418.981397] Key type .llcrypt registered [ 1419.039639] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1419.461793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1419.472971] alg: No test for adler32 (adler32-zlib) [ 1420.338620] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1420.436080] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1422.023210] Key type lgssc registered [ 1422.452978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.445763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.454374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.635089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.028317] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1431.186810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.305291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.630020] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1435.129351] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.123490] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1439.410236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.519350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.406631] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1443.854495] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.838117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1457.119732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1457.122110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.126819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.144868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1458.146052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.146978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.243295] Lustre: server umount lustre-OST0000 complete [ 1467.359945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1467.363171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.368314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.371603] Lustre: Skipped 1 previous similar message [ 1470.545921] Lustre: server umount lustre-MDT0000 complete [ 1471.696243] LustreError: 81386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123395 with bad export cookie 4412417579036131458 [ 1471.698399] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.700519] LustreError: 81386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1471.800770] Lustre: server umount lustre-MDT0001 complete [ 1475.113200] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1476.103113] Key type lgssc unregistered [ 1476.225367] LNet: 84957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.227607] LNetError: 84957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.235409] LNet: Removed LNI 192.168.202.135@tcp [ 1476.563699] Key type .llcrypt unregistered [ 1476.565151] Key type ._llcrypt unregistered [ 1484.907293] Key type ._llcrypt registered [ 1484.908688] Key type .llcrypt registered [ 1484.940482] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1485.276079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1485.321607] alg: No test for adler32 (adler32-zlib) [ 1486.167270] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1486.245379] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1487.823154] Key type lgssc registered [ 1488.148381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1490.616100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1490.622135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.748426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.963950] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1495.584741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.693327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1497.028922] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1499.380387] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1501.140581] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.327887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.436791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.256958] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1507.510151] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1516.773227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1522.145621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.146493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.149111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.154187] Lustre: Skipped 1 previous similar message [ 1526.342284] Lustre: server umount lustre-OST0000 complete [ 1531.872130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1531.874455] LustreError: Skipped 1 previous similar message [ 1531.875635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.879680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.882463] Lustre: Skipped 1 previous similar message [ 1533.656583] Lustre: server umount lustre-MDT0000 complete [ 1534.896318] LustreError: 86328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123458 with bad export cookie 8713471452798818119 [ 1534.899557] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.900241] LustreError: 86328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1535.020583] Lustre: server umount lustre-MDT0001 complete [ 1538.517928] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1539.618496] Key type lgssc unregistered [ 1539.747335] LNet: 89896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.749730] LNetError: 89896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.760293] LNet: Removed LNI 192.168.202.135@tcp [ 1540.071762] Key type .llcrypt unregistered [ 1540.073049] Key type ._llcrypt unregistered [ 1548.598114] Key type ._llcrypt registered [ 1548.599818] Key type .llcrypt registered [ 1548.635220] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1549.049486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1549.055959] alg: No test for adler32 (adler32-zlib) [ 1549.912371] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1549.995806] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1551.575142] Key type lgssc registered [ 1551.934154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1554.560216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1554.567376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.712334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.120317] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1560.011925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.160735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1561.654098] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1564.209559] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1566.080749] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1568.379508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1568.495498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.412234] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1572.789038] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1581.796477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1587.169869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.170974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.171673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.175786] Lustre: Skipped 1 previous similar message [ 1592.288406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1592.290448] Lustre: Skipped 2 previous similar messages [ 1592.401074] Lustre: server umount lustre-OST0000 complete [ 1595.872307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1595.875117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.879336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.409101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.760941] Lustre: server umount lustre-MDT0000 complete [ 1600.979214] LustreError: 91268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123524 with bad export cookie 7463105560830341301 [ 1600.982502] LustreError: 91268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1600.982665] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.092055] Lustre: server umount lustre-MDT0001 complete [ 1604.488285] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1605.439421] Key type lgssc unregistered [ 1605.568314] LNet: 94838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.570625] LNetError: 94838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.579311] LNet: Removed LNI 192.168.202.135@tcp [ 1605.876140] Key type .llcrypt unregistered [ 1605.877057] Key type ._llcrypt unregistered [ 1614.023675] Key type ._llcrypt registered [ 1614.025189] Key type .llcrypt registered [ 1614.062902] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1614.386087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1614.428816] alg: No test for adler32 (adler32-zlib) [ 1615.258314] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1615.339751] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1616.911105] Key type lgssc registered [ 1617.224358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.550976] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.557651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1634.975379] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1640.095301] LustreError: 96202:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 1666.719143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.648869] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1671.152105] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1672.973274] Lustre: server umount lustre-OST0000 complete [ 1675.538249] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:53:18 (1784123598) [ 1677.716018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.861718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.111133] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1681.775158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.880045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1683.098161] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1685.319447] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1687.063119] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1689.150746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.254493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.020542] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1693.258153] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1703.589916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1707.999743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.001583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.005101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1709.024678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1709.025788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1709.026939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.125166] Lustre: server umount lustre-OST0000 complete [ 1712.343860] Lustre: server umount lustre-MDT0000 complete [ 1712.607725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1712.610319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.615325] LustreError: 98564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.621095] LustreError: 98564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1713.491294] LustreError: 97820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123636 with bad export cookie 1261944524798897553 [ 1713.495185] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.495585] LustreError: 97820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1713.601666] Lustre: server umount lustre-MDT0001 complete [ 1716.858453] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1717.765923] Key type lgssc unregistered [ 1717.881338] LNet: 101544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.883622] LNetError: 101544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.894401] LNet: Removed LNI 192.168.202.135@tcp [ 1718.173880] Key type .llcrypt unregistered [ 1718.174914] Key type ._llcrypt unregistered [ 1731.716748] Key type ._llcrypt registered [ 1731.718025] Key type .llcrypt registered [ 1731.758372] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1732.063773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.069569] alg: No test for adler32 (adler32-zlib) [ 1732.919436] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1733.009794] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1734.583117] Key type lgssc registered [ 1734.889510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1737.264760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1737.269993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.394670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.584426] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1742.336481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.467167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.859362] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1746.313436] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.139825] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1750.257529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1750.359779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.096160] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1754.464572] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1762.662653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1767.905019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1767.905390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.906161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.908060] LustreError: Skipped 1 previous similar message [ 1767.912788] Lustre: Skipped 1 previous similar message [ 1769.425779] Lustre: server umount lustre-OST0000 complete [ 1772.755724] Lustre: server umount lustre-MDT0000 complete [ 1773.025433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.026160] LustreError: 103704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.030283] Lustre: Skipped 1 previous similar message [ 1773.035974] LustreError: 103704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1773.930265] LustreError: 103690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123697 with bad export cookie 2198444136519374445 [ 1773.933175] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.935676] LustreError: 103690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1774.037295] Lustre: server umount lustre-MDT0001 complete [ 1777.542185] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1778.501187] Key type lgssc unregistered [ 1778.624428] LNet: 107258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.627954] LNetError: 107258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.637409] LNet: Removed LNI 192.168.202.135@tcp [ 1778.953530] Key type .llcrypt unregistered [ 1778.955350] Key type ._llcrypt unregistered [ 1785.685160] Key type ._llcrypt registered [ 1785.686193] Key type .llcrypt registered [ 1785.723380] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1786.025405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.088582] alg: No test for adler32 (adler32-zlib) [ 1786.939842] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 1787.019962] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1788.599100] Key type lgssc registered [ 1788.916655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1791.232328] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1791.239590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.687342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1811.807290] LustreError: 108477:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 1838.431184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.303692] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1842.829080] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1845.061131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.156690] LustreError: MGC192.168.202.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1845.160935] LustreError: 109499:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1845.163363] LustreError: 109499:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1845.166536] LustreError: 109499:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1845.193844] Lustre: server umount lustre-MDT0000 complete [ 1845.195253] LustreError: 109499:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1858.047882] Lustre: server umount lustre-OST0000 complete [ 1861.031905] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 1864.145515] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1875.865534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.955610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1875.964746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1876.000123] Lustre: lustre-MDT0000: new disk, initializing [ 1876.020842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.028225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1877.364061] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1881.618512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.662588] Lustre: 112283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1881.687276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1881.689713] Lustre: Skipped 1 previous similar message [ 1881.728979] Lustre: lustre-MDT0001: new disk, initializing [ 1881.765053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1881.778368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1881.782192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1883.229410] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1886.998517] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1888.844919] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1891.090573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1891.194916] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1891.199491] Lustre: lustre-OST0000: new disk, initializing [ 1891.202342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1891.204675] Lustre: 114256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1891.235319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.131614] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1896.432443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1896.436562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1896.445038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1896.734465] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1898.113178] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1898.186753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1899.472895] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1899.541120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1901.535846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.538359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.542601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.584486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.585905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.586960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.270302] Lustre: server umount lustre-OST0000 complete [ 1908.705328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.709227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.713449] Lustre: Skipped 1 previous similar message [ 1908.715901] Lustre: Skipped 1 previous similar message [ 1913.827989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.838829] Lustre: Skipped 1 previous similar message [ 1914.309867] Lustre: server umount lustre-MDT0000 complete [ 1919.595618] LustreError: 112278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123842 with bad export cookie 17580757237932329691 [ 1919.601905] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.604300] LustreError: 112278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1919.916308] Lustre: server umount lustre-MDT0001 complete [ 1926.453712] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:57:28 (1784123848) [ 1929.372719] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1942.505472] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 1947.067425] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1962.718367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.819144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1962.830624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1962.868577] Lustre: lustre-MDT0000: new disk, initializing [ 1962.900552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.909413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1964.796751] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1969.251327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.296669] Lustre: 119146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1969.322945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1969.324795] Lustre: Skipped 1 previous similar message [ 1969.360794] Lustre: lustre-MDT0001: new disk, initializing [ 1969.419263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1969.423488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1970.992100] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1975.360309] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.163626] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1979.860426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1980.014366] Lustre: lustre-OST0000: new disk, initializing [ 1980.016830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1980.020540] Lustre: 121113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1980.058833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.064695] Lustre: Skipped 1 previous similar message [ 1982.562974] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1987.332517] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1988.610235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1988.617103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1988.673762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1989.065122] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1989.151627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1990.882783] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1990.972323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1993.699317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.702719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.707027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.711221] Lustre: Skipped 2 previous similar messages [ 1997.927469] Lustre: server umount lustre-OST0000 complete [ 2000.354464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.357699] LustreError: Skipped 1 previous similar message [ 2000.359230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.363339] Lustre: Skipped 1 previous similar message [ 2005.660374] Lustre: server umount lustre-MDT0000 complete [ 2007.135343] LustreError: 119140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123930 with bad export cookie 17580757237932330559 [ 2007.138726] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.140130] LustreError: 119140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2007.265267] Lustre: server umount lustre-MDT0001 complete [ 2010.586429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.813663] LustreError: 123325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.820935] LustreError: 123325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2010.851934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.532582] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2015.934826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.732463] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2020.293553] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.383823] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2025.081881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2027.142793] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2030.014210] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2043.646623] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2046.946331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2046.947183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.950561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.955364] Lustre: Skipped 2 previous similar messages [ 2046.957958] Lustre: Skipped 3 previous similar messages [ 2050.911630] Lustre: server umount lustre-OST0000 complete [ 2057.186590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2057.190059] LustreError: Skipped 1 previous similar message [ 2058.591306] Lustre: server umount lustre-MDT0000 complete [ 2060.213092] LustreError: 123307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784123983 with bad export cookie 17580757237932331427 [ 2060.217217] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2060.217844] LustreError: 123307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2060.360542] Lustre: server umount lustre-MDT0001 complete [ 2064.604641] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2065.935163] Key type lgssc unregistered [ 2066.102527] LNet: 126969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.105268] LNetError: 126969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.113368] LNet: Removed LNI 192.168.202.135@tcp [ 2066.463840] Key type .llcrypt unregistered [ 2066.465591] Key type ._llcrypt unregistered [ 2073.887038] Key type ._llcrypt registered [ 2073.888603] Key type .llcrypt registered [ 2073.937541] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 2080.894231] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2081.299180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2081.332874] alg: No test for adler32 (adler32-zlib) [ 2082.212110] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 2082.315601] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2083.903169] Key type lgssc registered [ 2084.360330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2098.959071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2098.965919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.054921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2100.076512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2100.114983] Lustre: lustre-MDT0000: new disk, initializing [ 2100.145759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.152968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2101.917698] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2107.207323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.255287] Lustre: 130095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.282213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2107.284544] Lustre: Skipped 1 previous similar message [ 2107.325425] Lustre: lustre-MDT0001: new disk, initializing [ 2107.399278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2107.428021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2107.433964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2109.262161] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2113.830937] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2116.136129] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2118.951039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.053471] Lustre: lustre-OST0000: new disk, initializing [ 2119.055529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2119.059561] Lustre: 132062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2119.094052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.096546] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2124.279741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2124.282829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2124.313395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2124.749024] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2126.163266] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2126.233680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2127.646231] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2127.735835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2129.378415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.378829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.384609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.388647] Lustre: Skipped 1 previous similar message [ 2130.589625] Lustre: server umount lustre-OST0000 complete [ 2132.959874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2132.962469] LustreError: Skipped 1 previous similar message [ 2132.964406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.968598] Lustre: Skipped 1 previous similar message [ 2132.970244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.496146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.496512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.616030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.618031] Lustre: Skipped 2 previous similar messages [ 2144.736226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.738512] Lustre: Skipped 1 previous similar message [ 2146.271121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2146.354529] Lustre: server umount lustre-MDT0000 complete [ 2147.832361] LustreError: 130087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124071 with bad export cookie 1147690189319175455 [ 2147.832975] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2147.838341] LustreError: 130087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2147.952674] Lustre: server umount lustre-MDT0001 complete [ 2150.921955] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:01:13 (1784124073) [ 2153.124576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.292467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.633211] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2157.667119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.793422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2159.154314] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2161.820374] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2163.750382] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2168.289874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.290841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.292683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2168.294808] Lustre: Skipped 1 previous similar message [ 2168.302786] Lustre: Skipped 1 previous similar message [ 2170.654287] Lustre: server umount lustre-MDT0000 complete [ 2171.996068] LustreError: 134467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124095 with bad export cookie 1147690189319176337 [ 2171.999102] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.001154] LustreError: 134467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2172.106348] Lustre: server umount lustre-MDT0001 complete [ 2174.692434] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:01:37 (1784124097) [ 2176.922653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2192.351617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2197.471459] LustreError: 136899:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 2223.071405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.205289] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2227.790367] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2228.526212] Lustre: server umount lustre-OST0000 complete [ 2231.669777] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:02:34 (1784124154) [ 2234.120505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.290083] LustreError: 138342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.297369] LustreError: 138342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2235.861912] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2239.410024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.458187] LustreError: 138343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2239.565902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.568457] Lustre: Skipped 1 previous similar message [ 2241.313761] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2244.235522] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.349177] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2248.956441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2251.290099] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2254.016225] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2269.665637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.666990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.670015] Lustre: Skipped 2 previous similar messages [ 2269.674556] Lustre: Skipped 4 previous similar messages [ 2273.240583] Lustre: server umount lustre-MDT0000 complete [ 2274.469199] LustreError: 138324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124197 with bad export cookie 1147690189319176988 [ 2274.471772] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.473647] LustreError: 138324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2274.577362] Lustre: server umount lustre-MDT0001 complete [ 2286.107402] Lustre: server umount lustre-OST0000 complete [ 2288.948535] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:03:31 (1784124211) [ 2291.357362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.532370] LustreError: 142010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.555734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.558431] Lustre: Skipped 1 previous similar message [ 2292.923742] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2295.699057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.172387] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2299.709279] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2301.665680] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2304.213248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.404953] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2308.981337] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2310.527560] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2316.647341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2316.725736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2318.234454] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2318.311604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2321.890919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.891161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2321.895105] Lustre: Skipped 1 previous similar message [ 2321.895784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.898671] LustreError: Skipped 1 previous similar message [ 2321.904144] Lustre: Skipped 1 previous similar message [ 2325.197424] Lustre: server umount lustre-OST0000 complete [ 2332.625420] Lustre: server umount lustre-MDT0000 complete [ 2333.889338] LustreError: 142744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124257 with bad export cookie 1147690189319178038 [ 2333.892456] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.894778] LustreError: 142744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2334.013616] Lustre: server umount lustre-MDT0001 complete [ 2336.764163] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:04:19 (1784124259) [ 2339.216778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2354.655527] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2359.775480] LustreError: 146206:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 2385.375303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.377935] Lustre: Skipped 2 previous similar messages [ 2387.228678] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2389.714264] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2392.081541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.228614] LustreError: 146231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.282354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2393.688995] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2396.897902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.625227] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2401.434394] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2403.357876] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2404.820478] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2404.903544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2406.393943] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2406.478810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2407.392070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2407.395877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.401531] Lustre: Skipped 2 previous similar messages [ 2407.404817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2407.410663] Lustre: Skipped 4 previous similar messages [ 2413.271933] Lustre: server umount lustre-MDT0000 complete [ 2414.541355] LustreError: 146213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124337 with bad export cookie 1147690189319179053 [ 2414.544311] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.545608] LustreError: 146213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2414.650234] Lustre: server umount lustre-MDT0001 complete [ 2426.271099] Lustre: server umount lustre-OST0000 complete [ 2429.353772] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:05:51 (1784124351) [ 2432.056993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.519805] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2452.639531] LustreError: 150392:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 2480.248985] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2482.756353] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2485.189081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.350406] LustreError: 150417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.397526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2486.939852] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2489.876863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.541086] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2494.356044] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2496.121816] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2498.474935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.550127] Lustre: lustre-OST0001: new disk, initializing [ 2498.552343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2498.556261] Lustre: 153264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2500.522157] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2504.184443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2504.187876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2504.219985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2504.275374] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2505.625324] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2505.702956] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2507.125464] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2507.210896] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2509.282812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.287783] Lustre: Skipped 3 previous similar messages [ 2513.979897] Lustre: server umount lustre-OST0000 complete [ 2514.405078] LustreError: 150409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.411458] LustreError: 150409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2519.522594] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2519.526179] LustreError: Skipped 1 previous similar message [ 2521.560198] Lustre: server umount lustre-OST0001 complete [ 2529.120298] Lustre: server umount lustre-MDT0000 complete [ 2530.604339] LustreError: 150399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124453 with bad export cookie 1147690189319180005 [ 2530.608351] LustreError: 150399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2530.608705] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.725454] Lustre: server umount lustre-MDT0001 complete [ 2539.669631] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:07:42 (1784124462) [ 2540.333077] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2540.927759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2541.508473] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:07:44 (1784124464) [ 2543.914463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.028163] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2544.040200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2544.118917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.122158] Lustre: Skipped 6 previous similar messages [ 2545.682888] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2548.906263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.953060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2548.964966] Lustre: 156817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2550.559380] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2553.826044] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2557.283654] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2562.871985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.995279] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2567.571292] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2572.903714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2574.630319] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2578.486192] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2578.670022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2583.095122] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2583.392450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2588.139381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.148831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.178444] Lustre: Skipped 4 previous similar messages [ 2588.185494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.189985] Lustre: Skipped 9 previous similar messages [ 2599.391214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2599.643491] Lustre: server umount lustre-OST0000 complete [ 2617.716148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2621.864943] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2625.694412] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2628.036469] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2629.287066] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.292663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.293759] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2629.305724] Lustre: Skipped 1 previous similar message [ 2630.227901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2632.193505] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2632.286077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2634.671404] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cbe04868000.ost_server_uuid 50 [ 2635.418530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8cbe04868000.ost_server_uuid in FULL state after 0 sec [ 2635.509506] LustreError: 156824:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.35@tcp) returned error from blocking AST (req@000000006749bb95 x1870789612864000 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000003ca719c2/0xfed6a257e921831 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.35@tcp remote: 0x6eb5f32686580bb6 expref: 5 pid: 156825 timeout: 2735 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2635.542259] LustreError: lustre-MDT0000: A client on nid 192.168.202.35@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2635.546071] LustreError: 156814:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.35@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000003ca719c2/0xfed6a257e921831 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.35@tcp remote: 0x6eb5f32686580bb6 expref: 6 pid: 156825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2639.843274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2639.846466] LustreError: Skipped 1 previous similar message [ 2643.354763] Lustre: server umount lustre-OST0000 complete [ 2651.556232] Lustre: server umount lustre-MDT0000 complete [ 2653.584956] LustreError: 157560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124576 with bad export cookie 1147690189319181188 [ 2653.586113] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2653.590491] LustreError: 157560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2659.885221] Lustre: server umount lustre-MDT0001 complete [ 2665.791240] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2667.653476] Key type lgssc unregistered [ 2667.836793] LNet: 162948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2667.840191] LNetError: 162948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2667.852346] LNet: Removed LNI 192.168.202.135@tcp [ 2668.429672] Key type .llcrypt unregistered [ 2668.431918] Key type ._llcrypt unregistered [ 2681.619930] Key type ._llcrypt registered [ 2681.621824] Key type .llcrypt registered [ 2681.688539] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2682.217230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2682.232337] alg: No test for adler32 (adler32-zlib) [ 2683.187422] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 2683.330209] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2684.935178] Key type lgssc registered [ 2685.491436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2689.681737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2689.703391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.935725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.434714] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2698.483954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.681853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2700.771792] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2704.179741] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2706.453556] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2709.231756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.371790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.644413] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2714.395348] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2716.523070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2727.079868] Lustre: Failing over lustre-MDT0000 [ 2727.214272] Lustre: server umount lustre-MDT0000 complete [ 2728.931149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.934445] LustreError: 164333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.938247] Lustre: Skipped 2 previous similar messages [ 2728.950484] LustreError: 164333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2734.049765] LustreError: 164335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.064320] LustreError: 164335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2738.319833] LustreError: 166935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.35@tcp arrived at 1784124661 with bad export cookie 5883174691396343763 [ 2738.328248] LustreError: 166935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2739.169777] LustreError: 165065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.176130] LustreError: 165065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2741.822911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.918509] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.068259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.120916] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2743.917904] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2744.965190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2744.969181] Lustre: lustre-MDT0000: Denying connection for new client 4c80ca7f-8605-47f6-bc66-4ba6c201f960 (at 192.168.202.35@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2747.375414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2752.496320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.500458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.504117] Lustre: Skipped 1 previous similar message [ 2757.601262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.605218] Lustre: Skipped 2 previous similar messages [ 2762.722559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.725659] Lustre: Skipped 1 previous similar message [ 2766.303196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2766.373168] Lustre: server umount lustre-OST0000 complete [ 2768.156306] LustreError: 168084:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2768.159730] Lustre: 167433:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2768.163926] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2768.173811] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2768.179558] LustreError: 167433:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a48988d6 x1870790242900736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2768.185168] LustreError: 167433:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2768.188359] LustreError: 167433:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2768.191927] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 2768.330720] Lustre: server umount lustre-MDT0000 complete [ 2770.316482] LustreError: 164319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124693 with bad export cookie 5883174691396344127 [ 2770.321715] LustreError: 165065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.323668] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.326610] LustreError: 164319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2776.578586] Lustre: server umount lustre-MDT0001 complete [ 2781.962723] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2783.257962] Key type lgssc unregistered [ 2783.420799] LNet: 168835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2783.425433] LNetError: 168835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2783.435545] LNet: Removed LNI 192.168.202.135@tcp [ 2783.843437] Key type .llcrypt unregistered [ 2783.845113] Key type ._llcrypt unregistered [ 2794.442669] Key type ._llcrypt registered [ 2794.443866] Key type .llcrypt registered [ 2794.493443] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2794.964486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2794.977708] alg: No test for adler32 (adler32-zlib) [ 2795.909129] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 2796.027860] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2797.639173] Key type lgssc registered [ 2798.138140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2801.466496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2801.475485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.749311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.309336] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2807.420581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.562318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.120760] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2812.532916] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2816.716188] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2822.678561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.899469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.082942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2828.381445] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2834.717379] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2842.595217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.603652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.611625] Lustre: Skipped 1 previous similar message [ 2847.747702] Lustre: server umount lustre-OST0000 complete [ 2852.832474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.840023] Lustre: Skipped 1 previous similar message [ 2852.842972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.846248] Lustre: Skipped 2 previous similar messages [ 2857.738660] Lustre: server umount lustre-MDT0000 complete [ 2858.017365] LustreError: 170243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.043346] LustreError: 170243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2861.000927] LustreError: 170205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124784 with bad export cookie 16992012846166736064 [ 2861.009593] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.230295] Lustre: server umount lustre-MDT0001 complete [ 2869.677407] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2872.444285] Key type lgssc unregistered [ 2872.777875] LNet: 173755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2872.783561] LNetError: 173755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.800103] LNet: Removed LNI 192.168.202.135@tcp [ 2873.608237] Key type .llcrypt unregistered [ 2873.612121] Key type ._llcrypt unregistered [ 2907.324869] Key type ._llcrypt registered [ 2907.326433] Key type .llcrypt registered [ 2907.406556] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2908.267144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2908.283816] alg: No test for adler32 (adler32-zlib) [ 2909.348207] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 2909.620439] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2911.265599] Key type lgssc registered [ 2912.275533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2918.003240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2918.016227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.313165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.874739] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2928.592119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.867251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.571566] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2936.294361] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2940.329665] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2945.261201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2945.490559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.635486] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2955.060397] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2964.994660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2974.178166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.181311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.196286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.202745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2975.205581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.215845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.234399] Lustre: Skipped 1 previous similar message [ 2976.835911] Lustre: server umount lustre-OST0000 complete [ 2980.325643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.326270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2980.341954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.354333] Lustre: Skipped 1 previous similar message [ 2985.456776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2985.462257] Lustre: Skipped 1 previous similar message [ 2985.751270] Lustre: server umount lustre-MDT0000 complete [ 2988.523331] LustreError: 175418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124911 with bad export cookie 2980192994328442777 [ 2988.529873] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.533498] LustreError: 175418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2988.751078] Lustre: server umount lustre-MDT0001 complete [ 2996.145652] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2998.155371] Key type lgssc unregistered [ 2998.324861] LNet: 178989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2998.328382] LNetError: 178989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2999.397223] LNet: Removed LNI 192.168.202.135@tcp [ 3000.119829] Key type .llcrypt unregistered [ 3000.122973] Key type ._llcrypt unregistered [ 3014.479662] Key type ._llcrypt registered [ 3014.481657] Key type .llcrypt registered [ 3014.541659] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3015.058440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3015.134579] alg: No test for adler32 (adler32-zlib) [ 3016.083947] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3016.238553] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3017.855417] Key type lgssc registered [ 3018.618343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3025.702778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3025.730381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.026107] Lustre: *** cfs_fail_loc=135, val=0*** [ 3027.075369] LustreError: 180490:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3027.079493] LustreError: 180490:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.135@tcp: cfg command failed: rc = -2 [ 3027.087678] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3027.087678] [ 3027.093943] LustreError: MGC192.168.202.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3027.106536] LustreError: 180452:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3027.110666] LustreError: 180452:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3027.113815] LustreError: 180452:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3027.121375] LustreError: 180452:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3027.163901] Lustre: server umount lustre-MDT0000 complete [ 3027.168396] LustreError: 180452:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3031.254710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.512046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.351445] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3040.296457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.507978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.900989] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3047.127536] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3050.437731] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3051.762024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.764839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3051.779378] Lustre: Skipped 1 previous similar message [ 3051.795400] Lustre: Skipped 1 previous similar message [ 3057.121240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.124444] Lustre: Skipped 1 previous similar message [ 3062.241652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.248735] Lustre: Skipped 1 previous similar message [ 3065.313280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.315753] Lustre: Skipped 1 previous similar message [ 3066.335129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3066.513852] Lustre: server umount lustre-MDT0000 complete [ 3068.759996] LustreError: 180843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784124992 with bad export cookie 2797283685183079160 [ 3068.761751] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.767230] LustreError: 180843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3068.924627] Lustre: server umount lustre-MDT0001 complete [ 3074.557860] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3076.192889] Key type lgssc unregistered [ 3076.393194] LNet: 183180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3076.397702] LNetError: 183180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3076.409237] LNet: Removed LNI 192.168.202.135@tcp [ 3076.929203] Key type .llcrypt unregistered [ 3076.934410] Key type ._llcrypt unregistered [ 3087.901135] Key type ._llcrypt registered [ 3087.902519] Key type .llcrypt registered [ 3087.951627] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3088.102300] Key type .llcrypt unregistered [ 3088.104368] Key type ._llcrypt unregistered [ 3098.014566] Key type ._llcrypt registered [ 3098.015918] Key type .llcrypt registered [ 3098.089680] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3098.766197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3098.799989] alg: No test for adler32 (adler32-zlib) [ 3099.761440] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3099.904962] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3101.495269] Key type lgssc registered [ 3102.288460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3107.209649] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3107.223522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.719614] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3127.841329] LustreError: 185289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 3154.463632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.840882] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3162.362615] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3166.448157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.783524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.832667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3168.710945] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3173.868272] LustreError: 186325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.885139] LustreError: 186325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3173.906882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.138622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3176.730704] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3180.862506] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.935561] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3185.731638] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3192.289612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.293551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.300833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.853717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.854138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3194.869525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.320740] Lustre: server umount lustre-OST0000 complete [ 3205.092630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.095032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.097288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.097302] Lustre: Skipped 1 previous similar message [ 3205.110027] Lustre: Skipped 1 previous similar message [ 3206.711918] Lustre: server umount lustre-MDT0000 complete [ 3208.966857] LustreError: 185296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125132 with bad export cookie 8126697601026409364 [ 3208.972534] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.979554] LustreError: 185296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3209.213477] Lustre: server umount lustre-MDT0001 complete [ 3215.915986] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3217.774323] Key type lgssc unregistered [ 3218.007435] LNet: 189200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.017579] LNetError: 189200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3218.041689] LNet: Removed LNI 192.168.202.135@tcp [ 3218.689630] Key type .llcrypt unregistered [ 3218.691706] Key type ._llcrypt unregistered [ 3234.494469] Key type ._llcrypt registered [ 3234.496656] Key type .llcrypt registered [ 3234.589549] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3235.294959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3235.306350] alg: No test for adler32 (adler32-zlib) [ 3236.263892] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3236.438538] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3238.048366] Key type lgssc registered [ 3238.688225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3243.503561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3243.527046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.817564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.403352] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3252.633190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.907035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3255.936630] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3260.491316] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3264.027245] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3268.297156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.516276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.575793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3272.492731] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3277.070622] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3281.154481] Lustre: Failing over lustre-MDT0000 [ 3281.317371] Lustre: server umount lustre-MDT0000 complete [ 3282.914809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.919671] LustreError: 190590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.934301] Lustre: Skipped 1 previous similar message [ 3282.954307] LustreError: 190590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3284.508344] LustreError: 190587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.032941] LustreError: 190591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.053228] LustreError: 190591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3293.155812] LustreError: 190585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.165694] LustreError: 190585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3297.324648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.428256] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.628105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.667610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3299.853167] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3300.312697] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3302.889680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3302.920646] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3302.955222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3303.156927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3304.447658] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3313.121498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.132455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.145554] Lustre: Skipped 1 previous similar message [ 3313.152488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.246141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.249277] Lustre: Skipped 1 previous similar message [ 3323.359198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3323.360944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3323.372824] Lustre: Skipped 2 previous similar messages [ 3323.434240] Lustre: server umount lustre-OST0000 complete [ 3328.483118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3328.483753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.492200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3328.514104] Lustre: Skipped 2 previous similar messages [ 3331.948576] Lustre: server umount lustre-MDT0000 complete [ 3333.600790] LustreError: 190591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.608932] LustreError: 190591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3334.259862] LustreError: 194267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125257 with bad export cookie 12580069698444278314 [ 3334.261691] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.270194] LustreError: 194267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3334.500814] Lustre: server umount lustre-MDT0001 complete [ 3341.090541] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3342.975471] Key type lgssc unregistered [ 3343.161060] LNet: 195324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3343.170769] LNetError: 195324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3343.192325] LNet: Removed LNI 192.168.202.135@tcp [ 3343.806745] Key type .llcrypt unregistered [ 3343.808377] Key type ._llcrypt unregistered [ 3360.846671] Key type ._llcrypt registered [ 3360.848775] Key type .llcrypt registered [ 3360.927386] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3361.625494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.638131] alg: No test for adler32 (adler32-zlib) [ 3362.557404] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3362.738980] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3364.377734] Key type lgssc registered [ 3365.204350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.480888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.508155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.808621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.494752] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3380.566359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.951868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3383.833338] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3388.388522] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3391.817383] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3395.805333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3396.055202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.100133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3399.852095] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3404.253557] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3407.306417] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3415.215611] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3426.358746] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3431.904618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3431.908361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.922089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3431.925490] Lustre: Skipped 1 previous similar message [ 3436.922157] Lustre: server umount lustre-OST0000 complete [ 3442.149043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.154248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.160053] Lustre: Skipped 2 previous similar messages [ 3447.264796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.270324] Lustre: Skipped 2 previous similar messages [ 3452.385315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3452.390326] Lustre: Skipped 1 previous similar message [ 3453.412690] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3453.510151] Lustre: server umount lustre-MDT0000 complete [ 3455.570000] LustreError: 196697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125378 with bad export cookie 6561515837825865327 [ 3455.572584] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3455.575241] LustreError: 196697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3455.743737] Lustre: server umount lustre-MDT0001 complete [ 3461.402584] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3462.980661] Key type lgssc unregistered [ 3463.155955] LNet: 200555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.162380] LNetError: 200555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3464.239466] LNet: Removed LNI 192.168.202.135@tcp [ 3464.700564] Key type .llcrypt unregistered [ 3464.702716] Key type ._llcrypt unregistered [ 3475.663167] Key type ._llcrypt registered [ 3475.664547] Key type .llcrypt registered [ 3475.727292] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3476.362384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.374521] alg: No test for adler32 (adler32-zlib) [ 3477.319867] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3477.472055] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3479.079210] Key type lgssc registered [ 3479.824777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3484.411194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3484.434708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.645359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.090805] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3493.086193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.290454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3495.736382] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3500.266716] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3503.223880] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3507.097116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3507.330442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.923631] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3511.485529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3515.909916] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3519.625702] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3528.238639] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3539.040996] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3544.844717] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3554.790344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.793352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.801589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3554.811019] Lustre: Skipped 1 previous similar message [ 3559.908145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3559.911873] Lustre: Skipped 1 previous similar message [ 3560.273067] Lustre: server umount lustre-OST0000 complete [ 3565.036320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.042246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3565.042468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.046500] Lustre: Skipped 2 previous similar messages [ 3568.818994] Lustre: server umount lustre-MDT0000 complete [ 3570.145434] LustreError: 201798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.155904] LustreError: 201798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3571.102530] LustreError: 201783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125494 with bad export cookie 15425297825625432615 [ 3571.105845] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3571.110157] LustreError: 201783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3571.319688] Lustre: server umount lustre-MDT0001 complete [ 3577.594990] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3579.456493] Key type lgssc unregistered [ 3579.640394] LNet: 206962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3579.649531] LNetError: 206962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3579.666170] LNet: Removed LNI 192.168.202.135@tcp [ 3580.230666] Key type .llcrypt unregistered [ 3580.232476] Key type ._llcrypt unregistered [ 3594.935620] Key type ._llcrypt registered [ 3594.937371] Key type .llcrypt registered [ 3595.004202] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3595.544224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3595.562374] alg: No test for adler32 (adler32-zlib) [ 3596.542101] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3596.699589] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3598.319339] Key type lgssc registered [ 3599.123327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3604.163929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3604.176522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.459489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.965162] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3613.227063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.480880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3616.101501] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3620.689509] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3623.941685] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.995898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.244800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.321050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3631.737066] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3636.269624] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3645.922983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3645.930163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.942908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3648.996970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.997145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3649.000370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.018652] Lustre: Skipped 1 previous similar message [ 3651.323442] Lustre: server umount lustre-OST0000 complete [ 3654.117984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.125094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3654.128468] Lustre: Skipped 1 previous similar message [ 3659.235323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.240691] Lustre: Skipped 2 previous similar messages [ 3659.699839] Lustre: server umount lustre-MDT0000 complete [ 3661.727822] LustreError: 208334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125585 with bad export cookie 12706928164366293378 [ 3661.730211] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.732940] LustreError: 208334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3661.889330] Lustre: server umount lustre-MDT0001 complete [ 3667.724162] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3669.365206] Key type lgssc unregistered [ 3669.547581] LNet: 212031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3669.557037] LNetError: 212031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3669.570103] LNet: Removed LNI 192.168.202.135@tcp [ 3670.092408] Key type .llcrypt unregistered [ 3670.094294] Key type ._llcrypt unregistered [ 3681.159603] Key type ._llcrypt registered [ 3681.161991] Key type .llcrypt registered [ 3681.231438] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3681.867787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3681.877305] alg: No test for adler32 (adler32-zlib) [ 3682.828983] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3682.991910] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3684.599188] Key type lgssc registered [ 3685.314349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3689.886685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3689.896201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.121154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.534188] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3698.403814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.656388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3701.213531] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3705.571750] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3708.829766] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3713.211527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3713.470041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.528229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3716.900382] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3721.562457] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3731.936121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3731.943563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.958931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3732.964657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3732.964848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.968287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.375867] Lustre: server umount lustre-OST0000 complete [ 3739.619769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3739.623357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3739.633792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3739.636109] Lustre: Skipped 1 previous similar message [ 3743.201375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3743.201464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.544336] Lustre: server umount lustre-MDT0000 complete [ 3746.499906] LustreError: 213257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125669 with bad export cookie 2416688660001636342 [ 3746.501800] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3746.504522] LustreError: 213257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3746.680964] Lustre: server umount lustre-MDT0001 complete [ 3752.638771] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3754.049707] Key type lgssc unregistered [ 3754.184578] LNet: 216954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3754.189652] LNetError: 216954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3754.201572] LNet: Removed LNI 192.168.202.135@tcp [ 3754.671056] Key type .llcrypt unregistered [ 3754.672876] Key type ._llcrypt unregistered [ 3764.735035] Key type ._llcrypt registered [ 3764.737032] Key type .llcrypt registered [ 3764.795247] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3765.378454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3765.393592] alg: No test for adler32 (adler32-zlib) [ 3766.316712] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3766.439948] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3768.055126] Key type lgssc registered [ 3768.622411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3772.403149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3772.413311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.622210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.428975] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3780.123980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.364190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3782.550455] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3786.655945] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3789.878051] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3793.718874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.933737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.244292] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3798.065222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3800.954155] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3806.731587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3806.791703] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3806.976319] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3809.962201] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3813.513477] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3814.295644] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3825.049467] Lustre: Permanently deactivating lustre-OST0001 [ 3825.056499] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3832.850142] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3835.882102] Lustre: Permanently reactivating lustre-OST0001 [ 3835.892407] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3835.899462] Lustre: Skipped 2 previous similar messages [ 3842.059938] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.072152] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3842.077478] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3842.084966] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3842.100921] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3847.137548] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3847.138742] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.145420] LustreError: Skipped 1 previous similar message [ 3847.154562] Lustre: Skipped 1 previous similar message [ 3847.156017] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3847.166791] Lustre: Skipped 1 previous similar message [ 3848.691425] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3848.696466] Lustre: Skipped 1 previous similar message [ 3849.955370] Lustre: server umount lustre-OST0001 complete [ 3852.256393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3852.260989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.266078] Lustre: Skipped 1 previous similar message [ 3852.267838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.798216] LustreError: 220019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.807291] LustreError: 220019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3858.011618] Lustre: server umount lustre-OST0000 complete [ 3862.495953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3862.499224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.503410] Lustre: Skipped 1 previous similar message [ 3862.506037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3862.509485] Lustre: Skipped 2 previous similar messages [ 3865.638779] Lustre: server umount lustre-MDT0000 complete [ 3867.232634] LustreError: 218178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784125790 with bad export cookie 9170494713722223621 [ 3867.236279] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.238093] LustreError: 218178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3867.391882] Lustre: server umount lustre-MDT0001 complete [ 3872.112937] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 3873.566369] Key type lgssc unregistered [ 3873.708876] LNet: 223239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3873.714431] LNetError: 223239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3873.726496] LNet: Removed LNI 192.168.202.135@tcp [ 3874.094584] Key type .llcrypt unregistered [ 3874.095827] Key type ._llcrypt unregistered [ 3892.138216] Key type ._llcrypt registered [ 3892.139754] Key type .llcrypt registered [ 3892.191765] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3892.702828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3892.725231] alg: No test for adler32 (adler32-zlib) [ 3893.609230] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 3893.726562] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 3895.327174] Key type lgssc registered [ 3895.788419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3899.119263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3899.128320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.240538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3900.250948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3900.314521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.087171] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3906.005348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.082573] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3906.100899] Lustre: 225417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3906.224577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.035299] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3911.218444] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3913.527139] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3916.312439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.391927] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3916.476464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.803321] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3919.136503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3921.915349] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3923.583100] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3929.222562] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3935.978145] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3945.750153] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3959.011691] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3959.014937] Lustre: Skipped 1 previous similar message [ 3978.016891] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3978.020310] Lustre: Skipped 2 previous similar messages [ 4014.191163] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4014.194378] Lustre: Skipped 4 previous similar messages [ 4078.049520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.053424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.054619] LustreError: Skipped 1 previous similar message [ 4078.071322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4078.075609] Lustre: Skipped 1 previous similar message [ 4083.171938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.177164] Lustre: Skipped 1 previous similar message [ 4083.694529] Lustre: server umount lustre-OST0000 complete [ 4085.729487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4085.736871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.742384] Lustre: Skipped 1 previous similar message [ 4085.745152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.288086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.288795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.561437] Lustre: server umount lustre-MDT0000 complete [ 4093.541221] LustreError: 228100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126016 with bad export cookie 12179969295449741535 [ 4093.544936] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.547159] LustreError: 228100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4093.731637] Lustre: server umount lustre-MDT0001 complete [ 4097.974875] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4099.090962] Key type lgssc unregistered [ 4099.225356] LNet: 230200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4099.227612] LNetError: 230200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4099.233549] LNet: Removed LNI 192.168.202.135@tcp [ 4099.552894] Key type .llcrypt unregistered [ 4099.554137] Key type ._llcrypt unregistered [ 4108.926935] Key type ._llcrypt registered [ 4108.928339] Key type .llcrypt registered [ 4108.968418] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4109.328077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4109.401636] alg: No test for adler32 (adler32-zlib) [ 4110.273929] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 4110.371936] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4111.951143] Key type lgssc registered [ 4112.322284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.262101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4115.279481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.453470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.183285] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4122.170227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.322295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4123.996467] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4126.828887] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4129.146166] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4131.818733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.977274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.567444] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4137.971215] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.226764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4151.776527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.778021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.779982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4151.784107] Lustre: Skipped 1 previous similar message [ 4154.837917] Lustre: server umount lustre-OST0000 complete [ 4156.896545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.902630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4156.904888] Lustre: Skipped 2 previous similar messages [ 4162.016328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.019502] Lustre: Skipped 1 previous similar message [ 4162.399465] Lustre: server umount lustre-MDT0000 complete [ 4163.971378] LustreError: 231571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784126087 with bad export cookie 13472341879291884972 [ 4163.979833] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.152111] Lustre: server umount lustre-MDT0001 complete [ 4168.508274] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4169.704848] Key type lgssc unregistered [ 4169.850681] LNet: 235275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4169.854752] LNetError: 235275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4169.867539] LNet: Removed LNI 192.168.202.135@tcp [ 4170.237249] Key type .llcrypt unregistered [ 4170.238832] Key type ._llcrypt unregistered [ 4180.298919] Key type ._llcrypt registered [ 4180.300223] Key type .llcrypt registered [ 4180.345656] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4180.682555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4180.725049] alg: No test for adler32 (adler32-zlib) [ 4181.591401] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 4181.696069] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4183.295173] Key type lgssc registered [ 4183.738916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4447.224136] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4448.282835] Key type lgssc unregistered [ 4448.403285] LNet: 237273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4448.406498] LNetError: 237273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4448.413343] LNet: Removed LNI 192.168.202.135@tcp [ 4448.722309] Key type .llcrypt unregistered [ 4448.723525] Key type ._llcrypt unregistered [ 4457.197192] Key type ._llcrypt registered [ 4457.198503] Key type .llcrypt registered [ 4457.235482] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 4463.502322] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4463.808986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4463.816896] alg: No test for adler32 (adler32-zlib) [ 4464.675413] Lustre: Lustre: Build Version: 2.17.54_162_g38d062c [ 4464.769061] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 4466.351159] Key type lgssc registered [ 4466.758280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4476.877923] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4325 sec ========== 10:39:59 (1784126399) [ 4477.446960] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:40:00 (1784126400) === [ 4478.559637] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:40:01 (1784126401) === [ 4492.663633] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 4493.733248] Key type lgssc unregistered [ 4493.858326] LNet: 242479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4493.860884] LNetError: 242479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4493.869340] LNet: Removed LNI 192.168.202.135@tcp [ 4494.170451] Key type .llcrypt unregistered [ 4494.172062] Key type ._llcrypt unregistered