[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494570016 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010088] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.021495] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026178] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] *** VALIDATE tmpfs *** [ 0.030381] *** VALIDATE proc *** [ 0.032244] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042030] debug: unmapping init [mem 0xffffffff86659000-0xffffffff86660fff] [ 0.044171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045652] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053280] rcu: Hierarchical SRCU implementation. [ 0.055388] smp: Bringing up secondary CPUs ... [ 0.056520] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.062012] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228808] node 0 deferred pages initialised in 162ms [ 0.233212] devtmpfs: initialized [ 0.234347] x86/mm: Memory block size: 128MB [ 0.238206] gcov: version magic: 0x41383552 [ 0.241340] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249433] pinctrl core: initialized pinctrl subsystem [ 0.252296] [ 0.253011] ************************************************************* [ 0.255021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258021] ** ** [ 0.261022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264020] ** ** [ 0.267021] ** This means that this kernel is built to expose internal ** [ 0.270021] ** IOMMU data structures, which may compromise security on ** [ 0.273020] ** your system. ** [ 0.276017] ** ** [ 0.279024] ** If you see this message and you are not debugging the ** [ 0.282018] ** kernel, report this immediately to your vendor! ** [ 0.284018] ** ** [ 0.287019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290018] ************************************************************* [ 0.292795] NET: Registered protocol family 16 [ 0.295520] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.299083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.303097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.307126] cpuidle: using governor menu [ 0.308636] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.309430] PCI: Using configuration type 1 for base access [ 0.310146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.317161] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322091] cryptd: max_cpu_qlen set to 1000 [ 0.324199] ACPI: Added _OSI(Module Device) [ 0.326013] ACPI: Added _OSI(Processor Device) [ 0.327016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.329011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334991] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.342554] ACPI: Interpreter enabled [ 0.344056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.345007] ACPI: Using IOAPIC for interrupt routing [ 0.346103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350570] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.361758] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.364063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.367026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.370179] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376685] acpiphp: Slot [2] registered [ 0.378283] acpiphp: Slot [5] registered [ 0.379219] acpiphp: Slot [6] registered [ 0.381244] acpiphp: Slot [7] registered [ 0.382162] acpiphp: Slot [8] registered [ 0.384165] acpiphp: Slot [9] registered [ 0.386240] acpiphp: Slot [10] registered [ 0.387168] acpiphp: Slot [3] registered [ 0.389096] acpiphp: Slot [4] registered [ 0.390158] acpiphp: Slot [11] registered [ 0.391106] acpiphp: Slot [12] registered [ 0.393121] acpiphp: Slot [13] registered [ 0.395126] acpiphp: Slot [14] registered [ 0.397105] acpiphp: Slot [15] registered [ 0.398104] acpiphp: Slot [16] registered [ 0.400097] acpiphp: Slot [17] registered [ 0.402099] acpiphp: Slot [18] registered [ 0.404096] acpiphp: Slot [19] registered [ 0.405119] acpiphp: Slot [20] registered [ 0.407095] acpiphp: Slot [21] registered [ 0.409088] acpiphp: Slot [22] registered [ 0.411115] acpiphp: Slot [23] registered [ 0.412088] acpiphp: Slot [24] registered [ 0.414118] acpiphp: Slot [25] registered [ 0.416107] acpiphp: Slot [26] registered [ 0.417111] acpiphp: Slot [27] registered [ 0.419107] acpiphp: Slot [28] registered [ 0.421093] acpiphp: Slot [29] registered [ 0.422098] acpiphp: Slot [30] registered [ 0.424118] acpiphp: Slot [31] registered [ 0.426151] PCI host bridge to bus 0000:00 [ 0.428029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.430034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.433096] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.436032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.439035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.442039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.443219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.447354] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.450526] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.461879] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.467018] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.470026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.472028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.475025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.478562] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.481963] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.487073] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.490737] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.496015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.510015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.515017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.521655] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.528023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.534023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.550022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.559581] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.572016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.589015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.602550] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.608019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.627023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.640224] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.653019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.669015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.679724] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.685018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.693020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.704022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.716834] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.726015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.733013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.748014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.765576] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.771327] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.774504] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.776494] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.779323] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.784131] iommu: Default domain type: Passthrough [ 0.786758] SCSI subsystem initialized [ 0.788213] ACPI: bus type USB registered [ 0.791127] usbcore: registered new interface driver usbfs [ 0.794083] usbcore: registered new interface driver hub [ 0.796090] usbcore: registered new device driver usb [ 0.798170] pps_core: LinuxPPS API ver. 1 registered [ 0.800011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.804056] PTP clock support registered [ 0.806118] EDAC MC: Ver: 3.0.0 [ 0.808178] PCI: Using ACPI for IRQ routing [ 0.809623] NetLabel: Initializing [ 0.811012] NetLabel: domain hash size = 128 [ 0.813010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.815088] NetLabel: unlabeled traffic allowed by default [ 0.818045] vgaarb: loaded [ 0.819182] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.821009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.827000] clocksource: Switched to clocksource kvm-clock [ 0.935501] VFS: Disk quotas dquot_6.6.0 [ 0.937344] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.940713] *** VALIDATE ramfs *** [ 0.942150] *** VALIDATE hugetlbfs *** [ 0.944023] pnp: PnP ACPI init [ 0.946522] pnp: PnP ACPI: found 6 devices [ 0.964810] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.968616] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.971155] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.974774] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.976896] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.979057] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.982048] NET: Registered protocol family 2 [ 0.984412] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.988855] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.992325] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.998141] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.001868] TCP: Hash tables configured (established 65536 bind 65536) [ 1.005393] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.008982] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.012191] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.015711] NET: Registered protocol family 1 [ 1.018412] RPC: Registered named UNIX socket transport module. [ 1.021825] RPC: Registered udp transport module. [ 1.024418] RPC: Registered tcp transport module. [ 1.026791] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.030089] NET: Registered protocol family 44 [ 1.032393] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.034858] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.037306] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.040128] PCI: CLS 0 bytes, default 64 [ 1.042782] Unpacking initramfs... [ 2.447907] debug: unmapping init [mem 0xffffa0eebcc54000-0xffffa0eebffbffff] [ 2.453970] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.456758] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.460221] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.962231] Initialise system trusted keyrings [ 2.965996] Key type blacklist registered [ 2.969432] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.979368] zbud: loaded [ 2.982721] *** VALIDATE nfs *** [ 2.984198] *** VALIDATE nfs4 *** [ 2.985892] pstore: using deflate compression [ 2.989996] Platform Keyring initialized [ 3.077074] NET: Registered protocol family 38 [ 3.079067] Key type asymmetric registered [ 3.080743] Asymmetric key parser 'x509' registered [ 3.082917] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.086754] io scheduler mq-deadline registered [ 3.088411] io scheduler kyber registered [ 3.090392] io scheduler bfq registered [ 3.092584] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.096151] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.099412] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.102613] ACPI: Power Button [PWRF] [ 3.108940] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.118284] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.136306] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.144956] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.162786] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.192421] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.221872] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.228339] Non-volatile memory driver v1.3 [ 3.230240] Linux agpgart interface v0.103 [ 3.269245] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.272143] vda: detected capacity change from 0 to 75071488 [ 3.291163] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.294391] vdb: detected capacity change from 0 to 1073741824 [ 3.309965] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.313183] vdc: detected capacity change from 0 to 2621440000 [ 3.330831] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.334534] vdd: detected capacity change from 0 to 2621440000 [ 3.348495] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.351692] vde: detected capacity change from 0 to 4294967296 [ 3.368430] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.371842] vdf: detected capacity change from 0 to 4294967296 [ 3.378827] libphy: Fixed MDIO Bus: probed [ 3.383689] usbcore: registered new interface driver usbserial_generic [ 3.386132] usbserial: USB Serial support registered for generic [ 3.388643] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.393816] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.396036] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.398846] mousedev: PS/2 mouse device common for all mice [ 3.403595] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.403656] rtc_cmos 00:05: RTC can wake from S4 [ 3.410972] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.411509] rtc_cmos 00:05: registered as rtc0 [ 3.417021] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.417425] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.424305] intel_pstate: CPU model not supported [ 3.427689] hid: raw HID events driver (C) Jiri Kosina [ 3.430292] usbcore: registered new interface driver usbhid [ 3.432492] usbhid: USB HID core driver [ 3.434344] drop_monitor: Initializing network drop monitor service [ 3.436986] Initializing XFRM netlink socket [ 3.439606] NET: Registered protocol family 10 [ 3.442567] Segment Routing with IPv6 [ 3.444417] NET: Registered protocol family 17 [ 3.447915] mpls_gso: MPLS GSO support [ 3.453221] RAS: Correctable Errors collector initialized. [ 3.455981] AVX version of gcm_enc/dec engaged. [ 3.458093] AES CTR mode by8 optimization enabled [ 3.548160] sched_clock: Marking stable (3548120278, 0)->(4470775885, -922655607) [ 3.553321] registered taskstats version 1 [ 3.556754] Loading compiled-in X.509 certificates [ 3.558974] zswap: loaded using pool lzo/zbud [ 3.588131] Key type big_key registered [ 3.603039] Key type encrypted registered [ 3.604653] ima: No TPM chip found, activating TPM-bypass! [ 3.606642] ima: Allocated hash algorithm: sha1 [ 3.608767] ima: No architecture policies found [ 3.610875] evm: Initialising EVM extended attributes: [ 3.612965] evm: security.selinux [ 3.613786] evm: security.ima [ 3.614469] evm: security.capability [ 3.615873] evm: HMAC attrs: 0x1 [ 3.618612] rtc_cmos 00:05: setting system clock to 2026-08-26 21:25:07 UTC (1787779507) [ 3.626553] debug: unmapping init [mem 0xffffffff87603000-0xffffffff877fffff] [ 3.629877] debug: unmapping init [mem 0xffffffff86382000-0xffffffff86658fff] [ 3.639073] Write protecting the kernel read-only data: 28672k [ 3.642685] debug: unmapping init [mem 0xffffffff84a03000-0xffffffff84bfffff] [ 3.645668] debug: unmapping init [mem 0xffffffff85314000-0xffffffff853fffff] [ 3.682466] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.689631] systemd[1]: Detected virtualization kvm. [ 3.691618] systemd[1]: Detected architecture x86-64. [ 3.693734] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.722385] systemd[1]: No hostname configured. [ 3.724439] systemd[1]: Set hostname to . [ 3.726801] random: systemd: uninitialized urandom read (16 bytes read) [ 3.729720] systemd[1]: Initializing machine ID from random generator. [ 3.781321] random: ln: uninitialized urandom read (6 bytes read) [ 3.890174] random: systemd: uninitialized urandom read (16 bytes read) [ 3.893561] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.900308] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.906339] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.631882] device-mapper: uevent: version 1.0.3 [ 4.634975] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 5.057613] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.528827] virtio_net virtio0 ens2: renamed from eth0 [ 5.590376] scsi host0: ata_piix [ 5.664871] scsi host1: ata_piix [ 5.666511] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.669228] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.433613] dracut-initqueue[595]: RTNETLINK answers: File exists [ 10.122213] random: crng init done [ 10.123955] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.690844] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.884526] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.136202] SELinux: Disabled at runtime. [ 12.200957] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.210246] systemd[1]: Detected virtualization kvm. [ 12.212722] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.661786] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.665955] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.671576] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.676639] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.681171] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.690514] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.695386] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ 12.865700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.163369] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.485784] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.512989] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.652478] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.663243] EDAC sbridge: Ver: 1.1.2 [ 15.204964] Key type dns_resolver registered [ 15.518654] NFS: Registering the id_resolver key type [ 15.520955] Key type id_resolver registered [ 15.522729] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg329-server login: [ 40.170160] libcfs: loading out-of-tree module taints kernel. [ 40.189547] Key type ._llcrypt registered [ 40.191306] Key type .llcrypt registered [ 40.241195] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 58.875951] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 60.314378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.323473] alg: No test for adler32 (adler32-zlib) [ 61.841858] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 62.710768] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 64.456184] Key type lgssc registered [ 66.059212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.856059] hrtimer: interrupt took 3066703 ns [ 81.736617] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.991796] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 137.196880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.271169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.640512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.679737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.798216] Lustre: lustre-MDT0000: new disk, initializing [ 138.933105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.954510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.448645] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.459827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.579492] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.629780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.633590] Lustre: Skipped 1 previous similar message [ 160.712858] Lustre: lustre-MDT0001: new disk, initializing [ 160.795317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.854877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.880262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.989432] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.528974] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.085118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.379474] Lustre: lustre-OST0000: new disk, initializing [ 180.386450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.393516] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.468593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.582045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 180.596596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 180.653613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 186.311624] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.307184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.509869] Lustre: lustre-OST0001: new disk, initializing [ 199.519101] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 199.530345] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.603801] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.376888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.386739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.442538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 206.799621] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.075388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.980570] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.385920] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.048070] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing yml_node [ 241.803508] Lustre: DEBUG MARKER: Client: 2.17.57.57 [ 244.079454] Lustre: DEBUG MARKER: MDS: 2.17.57.57 [ 246.354211] Lustre: DEBUG MARKER: OSS: 2.17.57.57 [ 247.864839] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 17:29:09 EDT 2026 [ 263.585582] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 264.993602] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 283.106628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.107807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.145517] Lustre: Skipped 1 previous similar message [ 286.402625] Lustre: server umount lustre-MDT0000 complete [ 289.520188] LustreError: 7467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787779793 with bad export cookie 18283284529843179733 [ 289.520811] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.534833] LustreError: 7467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 293.347446] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.353207] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.353910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.353916] Lustre: Skipped 3 previous similar messages [ 293.363723] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 293.393932] Lustre: Skipped 3 previous similar messages [ 294.367938] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.368288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.387873] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 294.416458] Lustre: Skipped 2 previous similar messages [ 295.967842] Lustre: server umount lustre-MDT0001 complete [ 299.812488] Lustre: server umount lustre-OST0000 complete [ 303.413278] Lustre: server umount lustre-OST0001 complete [ 309.113918] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 318.921284] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 356.721129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.001769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 357.036724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.103953] Lustre: lustre-MDT0000: new disk, initializing [ 357.236209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.252479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.778592] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 372.988856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.092749] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 373.146194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 373.152116] Lustre: Skipped 1 previous similar message [ 373.205712] Lustre: lustre-MDT0001: new disk, initializing [ 373.311584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 373.338405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 373.352925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 376.969237] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 385.795468] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.891074] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 399.070974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 399.347854] Lustre: lustre-OST0000: new disk, initializing [ 399.350429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 399.355271] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 399.436846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.470115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 400.475112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 400.574335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 406.356324] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 416.115619] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 422.374590] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 422.861389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.754500] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 429.901977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 436.705146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 436.710359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.725269] LustreError: Skipped 1 previous similar message [ 436.726050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.789543] Lustre: Skipped 1 previous similar message [ 438.898613] Lustre: server umount lustre-OST0000 complete [ 444.897143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.923877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.956684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.968247] Lustre: Skipped 1 previous similar message [ 450.707670] Lustre: server umount lustre-MDT0000 complete [ 452.064589] LustreError: 16645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.093184] LustreError: 16645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 454.975340] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787779958 with bad export cookie 18283284529843181266 [ 454.979503] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.996876] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 455.355067] Lustre: server umount lustre-MDT0001 complete [ 459.589248] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:32:40 (1787779960) [ 466.230771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.743540] LustreError: 19318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.808031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.340051] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 472.041343] LustreError: 19319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.170744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.890558] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 494.127633] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.358411] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.369944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.549620] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 526.207583] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 535.528812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 535.543278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.556704] Lustre: Skipped 1 previous similar message [ 535.561685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.564774] Lustre: Skipped 2 previous similar messages [ 540.644369] LustreError: 21140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.670331] LustreError: 21140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 540.775969] Lustre: server umount lustre-OST0000 complete [ 547.304278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 547.310201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.318728] Lustre: Skipped 1 previous similar message [ 552.540605] Lustre: server umount lustre-MDT0000 complete [ 557.759065] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787780061 with bad export cookie 18283284529843182148 [ 557.774197] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 557.776358] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.389404] Lustre: server umount lustre-MDT0001 complete [ 571.214658] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 573.965931] Key type lgssc unregistered [ 574.323506] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.330307] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.347074] LNet: Removed LNI 192.168.203.129@tcp [ 575.211429] Key type .llcrypt unregistered [ 575.215270] Key type ._llcrypt unregistered [ 599.718912] Key type ._llcrypt registered [ 599.720677] Key type .llcrypt registered [ 599.819530] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 601.228051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.244846] alg: No test for adler32 (adler32-zlib) [ 602.349943] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 602.649230] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 604.289290] Key type lgssc registered [ 605.429298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 613.317826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 613.345706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.934302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.341403] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 625.126396] LustreError: 24265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.146993] LustreError: 24265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 627.765610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.087403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.149354] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 639.516592] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 645.226681] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 653.017131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 653.634772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.624689] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 660.805701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 668.780689] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 686.565289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 686.576496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.591941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 686.919881] Lustre: server umount lustre-OST0000 complete [ 693.730162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.735085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 693.743264] Lustre: Skipped 1 previous similar message [ 697.400147] Lustre: server umount lustre-MDT0000 complete [ 698.849368] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.876707] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 701.465863] LustreError: 24247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787780205 with bad export cookie 6066925703922384283 [ 701.474518] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.478019] LustreError: 24247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 711.790250] Lustre: server umount lustre-MDT0001 complete [ 722.326232] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 725.198919] Key type lgssc unregistered [ 725.526160] LNet: 28248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.534822] LNetError: 28248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.558691] LNet: Removed LNI 192.168.203.129@tcp [ 726.306471] Key type .llcrypt unregistered [ 726.317748] Key type ._llcrypt unregistered [ 750.849935] Key type ._llcrypt registered [ 750.860037] Key type .llcrypt registered [ 751.047292] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 752.596446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 752.666191] alg: No test for adler32 (adler32-zlib) [ 753.867322] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 754.157382] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 755.839308] Key type lgssc registered [ 757.142161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 765.874273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 765.894169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.396870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.877261] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 777.704351] LustreError: 29662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.722373] LustreError: 29662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 781.738771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.140028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 787.752188] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 796.460248] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 803.559210] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 820.905578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 821.214373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.398257] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 836.093838] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 837.615801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 846.816318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 846.824447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.842562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 847.841178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.841952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 847.872302] Lustre: Skipped 1 previous similar message [ 850.098608] Lustre: server umount lustre-OST0000 complete [ 858.082110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.095892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.107969] Lustre: Skipped 1 previous similar message [ 858.120171] Lustre: Skipped 1 previous similar message [ 860.936926] Lustre: server umount lustre-MDT0000 complete [ 863.204630] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.225534] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 865.173920] LustreError: 32790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787780369 with bad export cookie 13063011819654398093 [ 865.185143] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.188390] LustreError: 32790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 865.750779] Lustre: server umount lustre-MDT0001 complete [ 877.959730] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 881.114830] Key type lgssc unregistered [ 881.384319] LNet: 33545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.392970] LNetError: 33545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.410272] LNet: Removed LNI 192.168.203.129@tcp [ 882.221438] Key type .llcrypt unregistered [ 882.228586] Key type ._llcrypt unregistered [ 906.325763] Key type ._llcrypt registered [ 906.334445] Key type .llcrypt registered [ 906.455590] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 907.688067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.764362] alg: No test for adler32 (adler32-zlib) [ 908.813623] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 909.032259] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 910.767102] Key type lgssc registered [ 912.041631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 919.775652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 919.800200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.251940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.285101] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 926.688264] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.707228] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 931.810653] LustreError: 34963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.289970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.700346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.753861] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 949.377789] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 955.779931] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 962.178287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 962.563184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.649099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 968.421567] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 977.318404] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 992.225277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.236701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.253219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.320399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.324310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.327414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.992700] Lustre: server umount lustre-OST0000 complete [ 1001.445876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.480162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.485312] Lustre: Skipped 2 previous similar messages [ 1006.561801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.569020] Lustre: Skipped 1 previous similar message [ 1011.681033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.696701] Lustre: Skipped 1 previous similar message [ 1015.263480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1015.612972] Lustre: server umount lustre-MDT0000 complete [ 1016.802743] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.825471] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1019.998950] LustreError: 34943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787780523 with bad export cookie 663981890558297863 [ 1020.006843] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.019276] LustreError: 34943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1020.650822] Lustre: server umount lustre-MDT0001 complete [ 1032.363578] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1035.526383] Key type lgssc unregistered [ 1035.802531] LNet: 38523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.813380] LNetError: 38523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.832647] LNet: Removed LNI 192.168.203.129@tcp [ 1036.921961] Key type .llcrypt unregistered [ 1036.926847] Key type ._llcrypt unregistered [ 1063.293278] Key type ._llcrypt registered [ 1063.296478] Key type .llcrypt registered [ 1063.411395] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1064.810850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.825672] alg: No test for adler32 (adler32-zlib) [ 1065.802322] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1066.082238] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1067.807177] Key type lgssc registered [ 1069.245796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.833755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.874194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.843833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.058412] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1088.995815] LustreError: 39938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.031239] LustreError: 39938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1094.137215] LustreError: 39939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.715733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.057437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1100.029258] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1107.160833] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1114.477317] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1123.489144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.752643] LustreError: 41759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.901504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.957323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1129.763735] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1137.655143] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.323748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.325622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.361412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.384530] Lustre: Skipped 1 previous similar message [ 1146.952930] Lustre: server umount lustre-OST0000 complete [ 1170.919452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.921760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.931935] Lustre: Skipped 2 previous similar messages [ 1170.956037] Lustre: Skipped 2 previous similar messages [ 1175.753029] Lustre: server umount lustre-MDT0000 complete [ 1176.040470] LustreError: 39939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.072801] LustreError: 39939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1179.730084] LustreError: 41781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787780683 with bad export cookie 5880270754936138832 [ 1179.735106] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.753088] LustreError: 41781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1186.230881] Lustre: server umount lustre-MDT0001 complete [ 1197.797459] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1201.034446] Key type lgssc unregistered [ 1201.490694] LNet: 43652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.499048] LNetError: 43652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.518654] LNet: Removed LNI 192.168.203.129@tcp [ 1202.618506] Key type .llcrypt unregistered [ 1202.624345] Key type ._llcrypt unregistered [ 1227.067450] Key type ._llcrypt registered [ 1227.069949] Key type .llcrypt registered [ 1227.192997] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1228.308740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.576959] alg: No test for adler32 (adler32-zlib) [ 1229.757530] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1230.079786] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1231.807154] Key type lgssc registered [ 1233.015166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.455724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.496486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.063803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.972504] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1251.325179] LustreError: 45069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.339637] LustreError: 45069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1254.799159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.156261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1260.085658] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1268.527886] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1275.956630] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1282.785575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.069446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.155678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1290.951390] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1300.006546] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1307.564946] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [ 1309.669199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.688229] Lustre: Skipped 2 previous similar messages [ 1309.692095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1309.702131] Lustre: Skipped 2 previous similar messages [ 1312.660403] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [ 1313.304737] Lustre: server umount lustre-MDT0000 complete [ 1314.788031] LustreError: 45064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.811585] LustreError: 45064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1317.166156] LustreError: 46912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787780821 with bad export cookie 1950390770736100728 [ 1317.170933] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.175323] LustreError: 46912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1317.775569] LustreError: 45065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.798920] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [ 1319.905478] LustreError: 47102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.907320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.895793] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [ 1322.906228] Lustre: Skipped 1 previous similar message [ 1323.496306] Lustre: server umount lustre-MDT0001 complete [ 1344.679934] Lustre: server umount lustre-OST0000 complete [ 1361.492795] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1364.794213] Key type lgssc unregistered [ 1365.219314] LNet: 48934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.232579] LNetError: 48934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.249564] LNet: Removed LNI 192.168.203.129@tcp [ 1366.104920] Key type .llcrypt unregistered [ 1366.107268] Key type ._llcrypt unregistered [ 1392.674802] Key type ._llcrypt registered [ 1392.679036] Key type .llcrypt registered [ 1392.761649] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1393.557949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.585556] alg: No test for adler32 (adler32-zlib) [ 1394.687328] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1394.964827] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1396.631184] Key type lgssc registered [ 1397.601384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1404.604911] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1404.639307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.257830] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1425.375723] LustreError: 50319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 1452.128692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.746738] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1467.658409] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1634.926837] Lustre: server umount lustre-OST0000 complete [ 1652.097463] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1654.614900] Key type lgssc unregistered [ 1654.917089] LNet: 51996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.927764] LNetError: 51996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.975657] LNet: Removed LNI 192.168.203.129@tcp [ 1656.712124] Key type .llcrypt unregistered [ 1656.716207] Key type ._llcrypt unregistered [ 1682.418483] Key type ._llcrypt registered [ 1682.421937] Key type .llcrypt registered [ 1682.510453] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1683.359392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.495852] alg: No test for adler32 (adler32-zlib) [ 1684.552932] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1684.790734] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1686.487182] Key type lgssc registered [ 1687.495440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1695.550874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1695.605187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.197905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.053703] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1707.515782] LustreError: 53412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.538794] LustreError: 53412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1711.166762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.526593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.372373] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1723.493741] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1728.914084] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.058412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.460307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.051932] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1742.652106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1749.523136] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1753.065189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.070402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.077730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.098721] Lustre: Skipped 1 previous similar message [ 1758.176520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.193496] Lustre: Skipped 1 previous similar message [ 1763.300910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.310640] Lustre: Skipped 1 previous similar message [ 1766.367167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.466970] Lustre: server umount lustre-OST0000 complete [ 1773.025655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1773.038298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.065355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.069206] Lustre: Skipped 1 previous similar message [ 1776.749847] Lustre: server umount lustre-MDT0000 complete [ 1778.657085] LustreError: 53409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1778.673336] LustreError: 53409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1780.153415] LustreError: 53395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781284 with bad export cookie 4457391336969465960 [ 1780.158155] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.166412] LustreError: 53395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1780.480643] Lustre: server umount lustre-MDT0001 complete [ 1789.881297] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1792.067966] Key type lgssc unregistered [ 1792.299983] LNet: 56945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.311396] LNetError: 56945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.331555] LNet: Removed LNI 192.168.203.129@tcp [ 1793.075288] Key type .llcrypt unregistered [ 1793.077052] Key type ._llcrypt unregistered [ 1814.945294] Key type ._llcrypt registered [ 1814.951840] Key type .llcrypt registered [ 1815.057715] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1816.130366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.147744] alg: No test for adler32 (adler32-zlib) [ 1817.176963] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1817.362823] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1819.007169] Key type lgssc registered [ 1820.041673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.428990] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.463951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1843.039640] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1848.159628] LustreError: 58332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 1874.848377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.818503] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1888.965023] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1896.298825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.393875] LustreError: 57668:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090d8f1b5 x1874624586451200/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1896.789606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.870784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1901.489191] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1902.050877] LustreError: 58358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.078517] LustreError: 58358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1907.177973] LustreError: 58357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.219430] LustreError: 58358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.244176] LustreError: 58358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1910.667205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.975358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.309220] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1924.784560] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.902219] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1936.872157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.874048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1936.900351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.123218] Lustre: server umount lustre-OST0000 complete [ 1966.569463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.576475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.592544] Lustre: Skipped 2 previous similar messages [ 1966.595560] Lustre: Skipped 1 previous similar message [ 1969.081281] Lustre: server umount lustre-MDT0000 complete [ 1971.681212] LustreError: 60362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.702144] LustreError: 60362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1973.454665] LustreError: 58340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781477 with bad export cookie 8399184731499298968 [ 1973.458273] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1973.462590] LustreError: 58340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1979.849956] Lustre: server umount lustre-MDT0001 complete [ 1990.192905] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1993.083963] Key type lgssc unregistered [ 1993.352539] LNet: 62084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.369940] LNetError: 62084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.402886] LNet: Removed LNI 192.168.203.129@tcp [ 1994.201844] Key type .llcrypt unregistered [ 1994.204325] Key type ._llcrypt unregistered [ 2018.145885] Key type ._llcrypt registered [ 2018.147991] Key type .llcrypt registered [ 2018.244538] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2019.850849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2019.925724] alg: No test for adler32 (adler32-zlib) [ 2021.024307] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2021.322873] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2023.008308] Key type lgssc registered [ 2024.041993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2032.154405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2032.174314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.549794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.483586] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2043.877285] LustreError: 63499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.898607] LustreError: 63499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2047.635465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.135936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.009533] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2060.323592] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2066.601157] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.817517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.188622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.276778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2080.708778] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2088.865720] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2107.900500] Lustre: server umount lustre-OST0000 complete [ 2112.994849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.996800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.011275] Lustre: Skipped 1 previous similar message [ 2113.024632] Lustre: Skipped 1 previous similar message [ 2118.112091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.118323] Lustre: Skipped 1 previous similar message [ 2120.275242] Lustre: server umount lustre-MDT0000 complete [ 2123.231920] Lustre: 62812:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787781611/real 1787781611] req@00000000d76ac6f8 x1874624800388224/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787781627 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2123.234431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.252462] Lustre: 62812:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2123.274655] LustreError: 63500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.290357] LustreError: 63500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2123.592309] LustreError: 63482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781627 with bad export cookie 1188902243403189776 [ 2123.597708] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2123.607754] LustreError: 63482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2124.144315] Lustre: server umount lustre-MDT0001 complete [ 2134.347760] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2137.066746] Key type lgssc unregistered [ 2137.321806] LNet: 67056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.333307] LNetError: 67056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.351120] LNet: Removed LNI 192.168.203.129@tcp [ 2138.143560] Key type .llcrypt unregistered [ 2138.149130] Key type ._llcrypt unregistered [ 2160.529621] Key type ._llcrypt registered [ 2160.532943] Key type .llcrypt registered [ 2160.624538] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2161.338292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2161.370917] alg: No test for adler32 (adler32-zlib) [ 2162.361172] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2162.550565] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2164.239335] Key type lgssc registered [ 2165.241864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2175.109953] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:01:16 (1787781676) [ 2182.399459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2182.413160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.870228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.196310] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2193.379116] LustreError: 68754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.395786] LustreError: 68754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2197.128461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.619541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2202.383364] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2210.743667] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2216.557766] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2223.693420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.073224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.243026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2230.470915] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2237.482915] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2246.090764] Lustre: Failing over lustre-MDT0000 [ 2246.451397] Lustre: server umount lustre-MDT0000 complete [ 2247.653079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.662505] LustreError: 68751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.691294] Lustre: Skipped 2 previous similar messages [ 2247.739037] LustreError: 68751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2249.140032] LustreError: 68749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.776426] LustreError: 68755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.799831] LustreError: 68755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2254.461626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.536536] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.635382] Lustre: *** cfs_fail_loc=135, val=0*** [ 2254.729729] LustreError: 71840:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2254.738819] LustreError: 71840:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -2 [ 2254.757845] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2254.757845] [ 2254.776141] LustreError: MGC192.168.203.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2254.793443] LustreError: 71829:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2254.806971] LustreError: 71829:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2254.815506] LustreError: 71829:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2254.826346] LustreError: 71829:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2254.880305] Lustre: server umount lustre-MDT0000 complete [ 2254.889640] LustreError: 71829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2257.889909] LustreError: 69482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.904809] LustreError: 69482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2259.821387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.920655] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.107837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.163864] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2262.932873] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2264.466532] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2265.574550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2265.619286] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2265.656811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2270.689542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.694058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2270.701949] Lustre: Skipped 1 previous similar message [ 2270.720500] Lustre: Skipped 1 previous similar message [ 2271.875329] Lustre: server umount lustre-OST0000 complete [ 2275.808428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.811948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.820967] Lustre: Skipped 1 previous similar message [ 2275.821985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.838368] Lustre: Skipped 1 previous similar message [ 2280.931480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.982863] Lustre: server umount lustre-MDT0000 complete [ 2286.001213] LustreError: 69489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781789 with bad export cookie 9719691624136124882 [ 2286.013542] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.016929] LustreError: 69489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2286.367864] Lustre: server umount lustre-MDT0001 complete [ 2297.062816] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2299.567463] Key type lgssc unregistered [ 2299.823830] LNet: 73622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.844775] LNetError: 73622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.867056] LNet: Removed LNI 192.168.203.129@tcp [ 2300.755059] Key type .llcrypt unregistered [ 2300.758493] Key type ._llcrypt unregistered [ 2324.551945] Key type ._llcrypt registered [ 2324.556033] Key type .llcrypt registered [ 2324.638296] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2325.578275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2325.723445] alg: No test for adler32 (adler32-zlib) [ 2326.777469] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2326.986019] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2328.647397] Key type lgssc registered [ 2329.714601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2337.036822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2337.063145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.536851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.462797] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2349.031920] LustreError: 75037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.067127] LustreError: 75037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2352.380316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.869500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2356.805632] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2364.450952] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2371.934880] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2378.740574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.328209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.428280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2385.638978] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2393.047939] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.928254] Lustre: Failing over lustre-MDT0000 [ 2403.240703] Lustre: server umount lustre-MDT0000 complete [ 2404.320954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2404.329847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.353119] LustreError: 75037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.338852] LustreError: 75033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.908291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2411.413269] LustreError: 75033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.451306] LustreError: 75033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2412.834677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.904972] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.980239] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2413.025152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2413.072748] LustreError: 78127:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2413.078345] LustreError: 78127:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -9 [ 2413.082812] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2413.082812] [ 2413.088478] LustreError: MGC192.168.203.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2413.101809] LustreError: 78115:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2413.107768] LustreError: 78115:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2413.117987] LustreError: 78115:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2413.122909] LustreError: 78115:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2413.140694] Lustre: server umount lustre-MDT0000 complete [ 2413.144146] LustreError: 78115:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2416.534283] LustreError: 77836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.755451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.922403] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.931163] LustreError: 78474:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.129@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000068edcc14) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2419.941895] LustreError: 75031:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2420.133175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.183818] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2421.656487] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2424.574324] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2425.360219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2425.394764] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2425.497956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2435.559824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.563792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.574065] Lustre: Skipped 2 previous similar messages [ 2435.584513] Lustre: Skipped 3 previous similar messages [ 2440.695151] LustreError: 76861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.736047] LustreError: 76861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2441.259041] Lustre: server umount lustre-OST0000 complete [ 2450.912377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.920478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.925686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.943053] Lustre: Skipped 1 previous similar message [ 2454.045345] Lustre: server umount lustre-MDT0000 complete [ 2458.026320] LustreError: 75018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787781961 with bad export cookie 13587298065448803673 [ 2458.034310] LustreError: 75018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2458.034835] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.309332] Lustre: server umount lustre-MDT0001 complete [ 2470.071840] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2473.051298] Key type lgssc unregistered [ 2473.351735] LNet: 79911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.357115] LNetError: 79911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.409938] LNet: Removed LNI 192.168.203.129@tcp [ 2475.288400] Key type .llcrypt unregistered [ 2475.292060] Key type ._llcrypt unregistered [ 2487.390426] Key type ._llcrypt registered [ 2487.393882] Key type .llcrypt registered [ 2487.482707] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2487.784712] Key type .llcrypt unregistered [ 2487.789668] Key type ._llcrypt unregistered [ 2510.712303] Key type ._llcrypt registered [ 2510.716560] Key type .llcrypt registered [ 2510.804767] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2511.857632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2512.036838] alg: No test for adler32 (adler32-zlib) [ 2513.110289] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2513.331777] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2514.991148] Key type lgssc registered [ 2516.002739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.722089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.737535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2525.273848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.693713] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2535.409666] LustreError: 81819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.436214] LustreError: 81819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2537.556344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.868955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2542.067289] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2549.656636] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2554.991584] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2561.850992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.148616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.701398] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2570.353187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2576.294989] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2586.082884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.092446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2586.093332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2586.108975] Lustre: Skipped 1 previous similar message [ 2590.987281] Lustre: server umount lustre-OST0000 complete [ 2596.329725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.356147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.375114] Lustre: Skipped 2 previous similar messages [ 2601.444224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.447019] Lustre: Skipped 1 previous similar message [ 2602.092526] Lustre: server umount lustre-MDT0000 complete [ 2606.379818] LustreError: 81798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782110 with bad export cookie 12817209872118596795 [ 2606.381187] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2606.397416] LustreError: 81798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2607.084775] Lustre: server umount lustre-MDT0001 complete [ 2620.238756] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2623.080987] Key type lgssc unregistered [ 2623.452806] LNet: 85355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.458860] LNetError: 85355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.482500] LNet: Removed LNI 192.168.203.129@tcp [ 2624.111452] Key type .llcrypt unregistered [ 2624.113241] Key type ._llcrypt unregistered [ 2646.643415] Key type ._llcrypt registered [ 2646.646408] Key type .llcrypt registered [ 2646.766422] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2647.586767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2647.636813] alg: No test for adler32 (adler32-zlib) [ 2648.609183] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2648.802091] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2650.482650] Key type lgssc registered [ 2651.398516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2658.625405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2658.643236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.276163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.636890] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2670.574429] LustreError: 86771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.609351] LustreError: 86771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2675.462542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.680992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2675.941202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.867218] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2686.980292] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2692.028187] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2697.988430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.321689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.150054] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2705.525320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2710.927559] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2719.654732] Lustre: server umount lustre-OST0000 complete [ 2726.394678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.402060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.402465] Lustre: Skipped 1 previous similar message [ 2731.492340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.500440] Lustre: Skipped 2 previous similar messages [ 2732.596491] Lustre: server umount lustre-MDT0000 complete [ 2735.752454] LustreError: 86752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782239 with bad export cookie 4373315156967652479 [ 2735.754156] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.767880] LustreError: 86752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2745.528614] Lustre: server umount lustre-MDT0001 complete [ 2755.418255] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2758.174263] Key type lgssc unregistered [ 2758.455497] LNet: 90307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.472850] LNetError: 90307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.506918] LNet: Removed LNI 192.168.203.129@tcp [ 2759.367848] Key type .llcrypt unregistered [ 2759.370782] Key type ._llcrypt unregistered [ 2782.498060] Key type ._llcrypt registered [ 2782.500735] Key type .llcrypt registered [ 2782.608889] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2783.765415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2783.798120] alg: No test for adler32 (adler32-zlib) [ 2784.946479] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2785.277474] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2787.064189] Key type lgssc registered [ 2788.665770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.545533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2797.594760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.261378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.858123] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2809.828040] LustreError: 91720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.849431] LustreError: 91720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2810.968449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.382803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2816.234199] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2823.297481] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2828.525923] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2834.609185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.995097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.340533] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2844.216583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2847.848210] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2859.491850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2859.500682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.517130] LustreError: Skipped 1 previous similar message [ 2859.557922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.200507] Lustre: server umount lustre-OST0000 complete [ 2864.609689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.631359] Lustre: Skipped 1 previous similar message [ 2864.637057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.648614] Lustre: Skipped 2 previous similar messages [ 2869.729321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2869.737503] Lustre: Skipped 1 previous similar message [ 2870.356465] Lustre: server umount lustre-MDT0000 complete [ 2873.680956] LustreError: 92454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782377 with bad export cookie 9830817259773466997 [ 2873.687203] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.699658] LustreError: 92454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2873.984950] Lustre: server umount lustre-MDT0001 complete [ 2882.837220] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2885.199143] Key type lgssc unregistered [ 2885.472152] LNet: 95253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2885.480712] LNetError: 95253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2885.502377] LNet: Removed LNI 192.168.203.129@tcp [ 2886.193191] Key type .llcrypt unregistered [ 2886.195211] Key type ._llcrypt unregistered [ 2907.402810] Key type ._llcrypt registered [ 2907.406861] Key type .llcrypt registered [ 2907.541259] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2908.543036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2908.559580] alg: No test for adler32 (adler32-zlib) [ 2909.668502] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2909.989993] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2911.735191] Key type lgssc registered [ 2912.923833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2920.763723] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2920.814973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.415506] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2941.535641] LustreError: 96647:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 2968.287219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.276416] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2981.112909] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2986.275614] Lustre: server umount lustre-OST0000 complete [ 2994.247167] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:14:56 (1787782496) [ 3000.739763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.162971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.723439] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3006.438620] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.463227] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3011.563396] LustreError: 98288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.574426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.937675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.491300] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3031.066647] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3038.022220] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3044.218121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.719778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.807950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3050.984441] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3059.674159] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3072.480315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3072.487129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.496606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3075.556099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3075.557845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3075.568910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.603432] Lustre: Skipped 1 previous similar message [ 3077.836546] Lustre: server umount lustre-OST0000 complete [ 3081.699137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3081.705439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.713377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.797659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.798626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.822832] Lustre: Skipped 1 previous similar message [ 3087.691424] Lustre: server umount lustre-MDT0000 complete [ 3090.912112] LustreError: 98284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.920505] LustreError: 98284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3091.425500] LustreError: 98268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782595 with bad export cookie 481784850660421444 [ 3091.440594] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.797822] Lustre: server umount lustre-MDT0001 complete [ 3102.026759] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3104.722733] Key type lgssc unregistered [ 3105.048947] LNet: 101979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3105.058472] LNetError: 101979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3105.086876] LNet: Removed LNI 192.168.203.129@tcp [ 3106.054446] Key type .llcrypt unregistered [ 3106.059963] Key type ._llcrypt unregistered [ 3144.944165] Key type ._llcrypt registered [ 3144.950496] Key type .llcrypt registered [ 3145.063630] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3146.178652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.219105] alg: No test for adler32 (adler32-zlib) [ 3147.300500] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 3147.581531] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3149.279155] Key type lgssc registered [ 3150.412720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3158.012355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3158.054767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.456211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.824834] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3164.642643] LustreError: 104165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.657873] LustreError: 104165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3169.762079] LustreError: 104166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.474710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.832557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3177.472376] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3184.814412] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3190.040336] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3197.109034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.413389] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3197.529406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.747657] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3205.619231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3211.707211] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3220.960606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3220.970990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.976226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3220.993030] Lustre: Skipped 1 previous similar message [ 3221.010547] Lustre: Skipped 1 previous similar message [ 3226.083079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3226.095380] Lustre: Skipped 1 previous similar message [ 3231.213778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.230880] Lustre: Skipped 1 previous similar message [ 3234.271701] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3234.542865] Lustre: server umount lustre-OST0000 complete [ 3239.403023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3239.428127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.455985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.444520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.833800] Lustre: server umount lustre-MDT0000 complete [ 3246.560630] LustreError: 106282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.595634] LustreError: 106282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3249.053325] LustreError: 104146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782752 with bad export cookie 7486704326314035049 [ 3249.057250] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.063661] LustreError: 104146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3249.485915] Lustre: server umount lustre-MDT0001 complete [ 3262.226660] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3265.266319] Key type lgssc unregistered [ 3265.644627] LNet: 107702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.653488] LNetError: 107702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3266.733846] LNet: Removed LNI 192.168.203.129@tcp [ 3267.736762] Key type .llcrypt unregistered [ 3267.738895] Key type ._llcrypt unregistered [ 3287.563943] Key type ._llcrypt registered [ 3287.568242] Key type .llcrypt registered [ 3287.724764] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3288.970067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3289.063445] alg: No test for adler32 (adler32-zlib) [ 3290.234267] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 3290.430762] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3292.119648] Key type lgssc registered [ 3293.422530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3301.501303] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3301.520892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.087490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3322.207659] LustreError: 108950:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 3348.895250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.092136] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3362.661119] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3370.251426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.522677] LustreError: MGC192.168.203.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3370.536729] LustreError: 109973:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3370.549578] LustreError: 109973:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3370.563103] LustreError: 109973:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3370.639171] Lustre: server umount lustre-MDT0000 complete [ 3370.641047] LustreError: 109973:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3388.895480] Lustre: 108286:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787782876/real 1787782876] req@0000000066d85b37 x1874626131004672/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787782892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3388.939161] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3388.968445] LustreError: 108962:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3409.492268] Lustre: server umount lustre-OST0000 complete [ 3418.676561] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3428.347982] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3465.157809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.498769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3465.541971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3465.630714] Lustre: lustre-MDT0000: new disk, initializing [ 3465.725943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.743415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3470.640471] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3481.664239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.806974] Lustre: 112768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3481.845622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3481.848586] Lustre: Skipped 1 previous similar message [ 3481.905292] Lustre: lustre-MDT0001: new disk, initializing [ 3481.985946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3482.046504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3482.066912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3486.650560] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3496.490826] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3502.593390] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3509.589834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.883342] Lustre: lustre-OST0000: new disk, initializing [ 3509.889206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3509.899615] Lustre: 114736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3509.996663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.731228] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3516.987960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3517.001221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3517.077243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3524.344236] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3528.540861] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3528.889620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3533.092870] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3533.271045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3537.891163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.896383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.906797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.909925] Lustre: Skipped 1 previous similar message [ 3537.927581] Lustre: Skipped 1 previous similar message [ 3541.778783] Lustre: server umount lustre-OST0000 complete [ 3548.136948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.139805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.169063] Lustre: Skipped 1 previous similar message [ 3552.674942] Lustre: server umount lustre-MDT0000 complete [ 3556.584641] LustreError: 114735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783060 with bad export cookie 5427870166194732442 [ 3556.586468] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.602553] LustreError: 114735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3556.956602] Lustre: server umount lustre-MDT0001 complete [ 3565.872760] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:24:27 (1787783067) [ 3568.927294] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3582.619536] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3591.956965] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3626.960708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.254839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3627.321873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3627.416436] Lustre: lustre-MDT0000: new disk, initializing [ 3627.513518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.536325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3631.696267] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3642.063988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.210801] Lustre: 119634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3642.277088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3642.291622] Lustre: Skipped 1 previous similar message [ 3642.380422] Lustre: lustre-MDT0001: new disk, initializing [ 3642.479720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.524113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.542144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3646.742014] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3656.156713] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3662.171889] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3669.253702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3669.582175] Lustre: lustre-OST0000: new disk, initializing [ 3669.585393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3669.595641] Lustre: 121603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3669.682863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.257097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3675.264683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3675.419492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3675.533823] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3684.167990] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3689.178328] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3689.338378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3693.274074] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3693.491917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3696.100708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.100871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.109054] LustreError: Skipped 2 previous similar messages [ 3696.112674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.128859] Lustre: Skipped 1 previous similar message [ 3696.142883] Lustre: Skipped 2 previous similar messages [ 3701.230819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.358240] Lustre: server umount lustre-OST0000 complete [ 3706.337853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.352574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.359627] Lustre: Skipped 2 previous similar messages [ 3711.808775] Lustre: server umount lustre-MDT0000 complete [ 3715.250381] LustreError: 120439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783219 with bad export cookie 5427870166194733324 [ 3715.257868] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.267563] LustreError: 120439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3715.708491] Lustre: server umount lustre-MDT0001 complete [ 3724.216143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.749258] LustreError: 123819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.770301] LustreError: 123819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3724.817405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.316373] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3729.892151] LustreError: 123820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.015944] LustreError: 123819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.016899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.513586] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3749.478068] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3754.883625] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3760.442881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3766.417996] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3773.191574] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3779.631560] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3784.674813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.690447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.701409] Lustre: Skipped 1 previous similar message [ 3784.705058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3784.713952] Lustre: Skipped 2 previous similar messages [ 3789.765273] Lustre: server umount lustre-OST0000 complete [ 3794.403888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3794.421535] LustreError: Skipped 1 previous similar message [ 3794.427719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.440106] Lustre: Skipped 1 previous similar message [ 3799.544049] Lustre: server umount lustre-MDT0000 complete [ 3801.057030] LustreError: 123815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.095488] LustreError: 123815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3803.308275] LustreError: 123799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783307 with bad export cookie 5427870166194734199 [ 3803.310891] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.328524] LustreError: 123799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3803.794906] Lustre: server umount lustre-MDT0001 complete [ 3816.065991] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3819.067969] Key type lgssc unregistered [ 3819.349227] LNet: 127451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3819.359044] LNetError: 127451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3819.378227] LNet: Removed LNI 192.168.203.129@tcp [ 3820.253414] Key type .llcrypt unregistered [ 3820.254996] Key type ._llcrypt unregistered [ 3838.877343] Key type ._llcrypt registered [ 3838.880794] Key type .llcrypt registered [ 3838.986735] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3852.329083] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3853.385646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3853.538344] alg: No test for adler32 (adler32-zlib) [ 3854.600121] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 3854.895231] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3856.527275] Key type lgssc registered [ 3857.685129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.158271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3895.207868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.502841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3896.550766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3896.666991] Lustre: lustre-MDT0000: new disk, initializing [ 3896.760054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.779482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3899.890831] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3911.211836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.331661] Lustre: 130592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3911.381107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3911.385133] Lustre: Skipped 1 previous similar message [ 3911.448917] Lustre: lustre-MDT0001: new disk, initializing [ 3911.526392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.568553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3911.589181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3916.195062] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3926.269309] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3932.309756] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.972675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.235992] Lustre: lustre-OST0000: new disk, initializing [ 3939.243267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3939.252649] Lustre: 132561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.340492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.248980] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3948.579705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3948.594139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.682530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3953.320187] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3957.678714] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3957.928767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3962.057529] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3962.222419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3964.387356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3964.396655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.413805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3968.996459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3969.001093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.112881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.125122] Lustre: Skipped 2 previous similar messages [ 3978.207170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3978.370060] Lustre: server umount lustre-OST0000 complete [ 3983.331141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3983.339160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.355273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3987.749918] Lustre: server umount lustre-MDT0000 complete [ 3990.878974] LustreError: 130585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783494 with bad export cookie 2535893824498613746 [ 3990.884731] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.892191] LustreError: 130585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3991.171270] Lustre: server umount lustre-MDT0001 complete [ 3997.739882] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:31:39 (1787783499) [ 4003.580377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.022885] LustreError: 134975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.052176] LustreError: 134975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4004.102993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.044370] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4009.452650] LustreError: 134976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.549252] LustreError: 134975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.885617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.207073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4021.404448] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4027.478281] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4031.683924] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4033.505965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.522872] Lustre: Skipped 1 previous similar message [ 4033.530302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4033.537085] Lustre: Skipped 3 previous similar messages [ 4039.761909] Lustre: server umount lustre-MDT0000 complete [ 4042.722551] LustreError: 134971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.751161] LustreError: 134971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4043.715343] LustreError: 134955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783547 with bad export cookie 2535893824498614628 [ 4043.726352] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.736758] LustreError: 134955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4044.009948] Lustre: server umount lustre-MDT0001 complete [ 4052.524884] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:32:34 (1787783554) [ 4059.660162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4075.298557] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4080.416460] LustreError: 137389:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 4106.079812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.434631] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4119.196700] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4121.328995] Lustre: server umount lustre-OST0000 complete [ 4128.469772] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:33:50 (1787783630) [ 4133.782495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.107542] LustreError: 138834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.164785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.870552] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4143.584778] LustreError: 138835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.602518] LustreError: 138835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4146.168422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.356940] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4157.275743] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4161.791541] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4168.206780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.554660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.564649] Lustre: Skipped 1 previous similar message [ 4173.899522] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4180.616192] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4187.622545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4187.632419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.653539] Lustre: Skipped 1 previous similar message [ 4187.662834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.675694] Lustre: Skipped 2 previous similar messages [ 4193.489139] Lustre: server umount lustre-MDT0000 complete [ 4194.274679] LustreError: 138830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.287300] LustreError: 138830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4197.159355] LustreError: 138816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783701 with bad export cookie 2535893824498615293 [ 4197.161625] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4197.173739] LustreError: 138816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4197.544970] Lustre: server umount lustre-MDT0001 complete [ 4210.389846] Lustre: server umount lustre-OST0000 complete [ 4217.173429] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:35:19 (1787783719) [ 4222.631849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.791195] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4228.582285] LustreError: 142487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.599386] LustreError: 142487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4234.349277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.633446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4234.641531] Lustre: Skipped 1 previous similar message [ 4238.764175] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4245.214684] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4250.050498] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4255.705901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4261.365522] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4267.376480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4268.043059] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4272.247673] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4272.459694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4275.854964] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4276.017732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4277.730402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4277.731204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.754891] Lustre: Skipped 2 previous similar messages [ 4277.762263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.766897] Lustre: Skipped 4 previous similar messages [ 4283.698350] Lustre: server umount lustre-OST0000 complete [ 4292.685528] Lustre: server umount lustre-MDT0000 complete [ 4293.093170] LustreError: 144834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.120123] LustreError: 144834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4295.994249] LustreError: 142468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783799 with bad export cookie 2535893824498616357 [ 4295.998546] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4296.006287] LustreError: 142468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4296.350915] Lustre: server umount lustre-MDT0001 complete [ 4303.466491] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:36:45 (1787783805) [ 4308.753696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.319887] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4329.439469] LustreError: 146643:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 4359.490449] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4365.978459] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4371.860755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4372.320480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4372.323384] Lustre: Skipped 2 previous similar messages [ 4372.372573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4376.520174] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4384.093246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.532947] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4393.774336] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4398.493547] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4402.563064] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4402.806990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4407.034564] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4407.242316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4409.824497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4409.831626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.839317] Lustre: Skipped 3 previous similar messages [ 4409.846743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4409.851250] Lustre: Skipped 4 previous similar messages [ 4423.135130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4423.336649] Lustre: server umount lustre-MDT0000 complete [ 4426.210955] LustreError: 147680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4426.241416] LustreError: 147680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4427.082977] LustreError: 146649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783930 with bad export cookie 2535893824498617386 [ 4427.090821] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.099764] LustreError: 146649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4427.626168] Lustre: server umount lustre-MDT0001 complete [ 4440.428676] Lustre: server umount lustre-OST0000 complete [ 4447.550403] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:39:09 (1787783949) [ 4453.260485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4468.769366] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4473.889096] LustreError: 150838:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 4505.887804] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4512.783205] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4518.793118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.343780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4524.186893] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4532.870047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.082166] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4545.028739] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4551.850309] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4560.702646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4561.024314] Lustre: lustre-OST0001: new disk, initializing [ 4561.027945] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4561.034061] Lustre: 153710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4568.676548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4568.686662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4568.862729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4570.422582] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4580.332906] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4585.285511] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4585.497238] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4590.408896] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4590.714270] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4594.672199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.699975] Lustre: Skipped 2 previous similar messages [ 4594.726743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4594.738587] Lustre: Skipped 10 previous similar messages [ 4594.743640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4598.960782] Lustre: server umount lustre-OST0000 complete [ 4609.855797] Lustre: server umount lustre-OST0001 complete [ 4620.815343] Lustre: server umount lustre-MDT0000 complete [ 4624.497500] LustreError: 150845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784128 with bad export cookie 2535893824498618338 [ 4624.510195] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.513583] LustreError: 150845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4624.871687] Lustre: server umount lustre-MDT0001 complete [ 4647.764659] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:42:29 (1787784149) [ 4649.206730] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4650.959519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4652.686721] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:42:34 (1787784154) [ 4658.473639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.741523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4658.768107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4658.995280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.005494] Lustre: Skipped 5 previous similar messages [ 4663.389449] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4671.501736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.616966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4671.646234] Lustre: 157273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.185147] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4684.565742] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4690.392659] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4697.372571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.552222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4704.750027] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4705.515901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4713.770571] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4718.494954] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4718.712419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4724.255583] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4724.446775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4731.367819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.375807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4731.394666] Lustre: Skipped 6 previous similar messages [ 4731.406424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4731.412659] Lustre: Skipped 7 previous similar messages [ 4731.435517] LustreError: Skipped 1 previous similar message [ 4732.728611] Lustre: server umount lustre-OST0000 complete [ 4754.595168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4760.465535] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4768.232889] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4771.179434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.194226] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.194630] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4771.223027] Lustre: Skipped 1 previous similar message [ 4772.607581] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4772.760360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4776.574703] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4776.752631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4782.042785] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96610604b000.ost_server_uuid 50 [ 4783.546697] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96610604b000.ost_server_uuid in FULL state after 0 sec [ 4783.742375] LustreError: 157278:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.29@tcp) returned error from blocking AST (req@00000000226030dd x1874626722657152 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000001139d516/0x23314dff6558d93c lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.29@tcp remote: 0xba6b08b8d6bd1d44 expref: 5 pid: 157279 timeout: 4883 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4783.783959] LustreError: lustre-MDT0000: A client on nid 192.168.203.29@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4783.790903] LustreError: 157268:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.29@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000001139d516/0x23314dff6558d93c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.29@tcp remote: 0xba6b08b8d6bd1d44 expref: 6 pid: 157279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4789.735323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4794.156966] Lustre: server umount lustre-OST0000 complete [ 4804.110414] Lustre: server umount lustre-MDT0000 complete [ 4807.139125] LustreError: 158378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4807.165882] LustreError: 158378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4808.126230] LustreError: 157265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784312 with bad export cookie 2535893824498619521 [ 4808.138911] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4808.143019] LustreError: 157265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4814.704722] Lustre: server umount lustre-MDT0001 complete [ 4825.917079] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4828.749392] Key type lgssc unregistered [ 4829.029369] LNet: 163390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4829.037619] LNetError: 163390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4829.054654] LNet: Removed LNI 192.168.203.129@tcp [ 4829.853843] Key type .llcrypt unregistered [ 4829.858663] Key type ._llcrypt unregistered [ 4857.381545] Key type ._llcrypt registered [ 4857.383394] Key type .llcrypt registered [ 4857.472488] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4858.370684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4858.408436] alg: No test for adler32 (adler32-zlib) [ 4859.668380] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 4859.876472] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4861.585975] Key type lgssc registered [ 4862.730881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4869.773224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4869.811460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.391088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.400722] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4881.893299] LustreError: 164805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.918433] LustreError: 164805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4883.442074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.817202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4887.939619] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4896.758224] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4903.784469] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4910.795590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4911.210191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4916.341198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4917.748693] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4926.780784] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4933.956982] Lustre: Failing over lustre-MDT0000 [ 4934.126959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.132148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4934.147315] Lustre: Skipped 1 previous similar message [ 4934.564485] LustreError: 164802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.672632] Lustre: server umount lustre-MDT0000 complete [ 4939.235903] LustreError: 166364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4939.264430] LustreError: 166364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4944.357628] LustreError: 164805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.380020] LustreError: 164805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4946.846235] LustreError: 164784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.29@tcp arrived at 1787784450 with bad export cookie 7937940298720233940 [ 4949.473778] LustreError: 166364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.492599] LustreError: 166364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4954.224366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.405239] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4954.602830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4954.616231] Lustre: Skipped 4 previous similar messages [ 4954.723769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.779079] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4959.729386] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4959.732829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4959.824900] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4962.252615] Lustre: lustre-MDT0000: Denying connection for new client c29fe1d2-135e-4eab-a3c3-ee79ec376985 (at 192.168.203.29@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 4975.080213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4975.081884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4975.082291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4975.097539] Lustre: Skipped 2 previous similar messages [ 4976.387202] Lustre: server umount lustre-OST0000 complete [ 4979.385741] LustreError: 168535:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4979.393435] Lustre: 167883:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4979.404247] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4979.422056] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4979.434719] LustreError: 167883:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000fb2779c x1874627776261632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4979.456351] LustreError: 167883:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4979.473270] LustreError: 167883:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4979.490267] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 4979.765961] Lustre: server umount lustre-MDT0000 complete [ 4980.194184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.221543] LustreError: 164802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4983.328370] LustreError: 164784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784487 with bad export cookie 7937940298720234045 [ 4983.335950] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4983.362227] LustreError: 164784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4985.887900] Lustre: server umount lustre-MDT0001 complete [ 4995.440997] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4997.806503] Key type lgssc unregistered [ 4998.052721] LNet: 169291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4998.059528] LNetError: 169291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4998.076622] LNet: Removed LNI 192.168.203.129@tcp [ 4998.856303] Key type .llcrypt unregistered [ 4998.858274] Key type ._llcrypt unregistered [ 5020.582845] Key type ._llcrypt registered [ 5020.585212] Key type .llcrypt registered [ 5020.695889] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5021.964791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5022.028180] alg: No test for adler32 (adler32-zlib) [ 5023.130592] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5023.383087] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5025.047160] Key type lgssc registered [ 5026.103895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5034.228142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5034.246283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.724857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.037924] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5045.218250] LustreError: 170712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.235609] LustreError: 170712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5048.238587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.709581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5052.897640] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5060.786743] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5066.661937] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5073.214739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.381242] LustreError: 172531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.610095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.097262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5081.337546] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5089.976427] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5097.147346] Lustre: server umount lustre-OST0000 complete [ 5101.543053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.550428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5101.566603] Lustre: Skipped 1 previous similar message [ 5106.658032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5106.668206] Lustre: Skipped 2 previous similar messages [ 5111.776470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5111.781475] Lustre: Skipped 1 previous similar message [ 5114.335116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5114.482536] Lustre: server umount lustre-MDT0000 complete [ 5115.888435] Lustre: 170024:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784603/real 1787784603] req@0000000022f39557 x1874627948210688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787784619 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5115.927142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.957300] LustreError: 170713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.975637] LustreError: 170713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5117.983140] Lustre: 170024:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784605/real 1787784605] req@00000000c1adde6c x1874627948212480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787784621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5118.200468] LustreError: 170691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784622 with bad export cookie 3133511637654445115 [ 5118.224224] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.515401] Lustre: server umount lustre-MDT0001 complete [ 5128.390217] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5130.783872] Key type lgssc unregistered [ 5131.063868] LNet: 174246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5131.068885] LNetError: 174246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5131.092615] LNet: Removed LNI 192.168.203.129@tcp [ 5131.951437] Key type .llcrypt unregistered [ 5131.954297] Key type ._llcrypt unregistered [ 5163.903312] Key type ._llcrypt registered [ 5163.911403] Key type .llcrypt registered [ 5164.014875] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5165.158851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5165.168344] alg: No test for adler32 (adler32-zlib) [ 5166.225494] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5166.550478] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5168.223637] Key type lgssc registered [ 5169.889682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5177.812540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5177.834600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.267111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.245598] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5189.605191] LustreError: 175957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5189.628226] LustreError: 175957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5193.654904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.034262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5198.654730] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5206.413951] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5211.710410] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5218.253762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5218.615308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.143484] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5224.753929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5231.883082] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5240.290632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.291886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.310364] LustreError: Skipped 1 previous similar message [ 5240.311792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5240.355551] Lustre: Skipped 1 previous similar message [ 5245.459736] Lustre: server umount lustre-OST0000 complete [ 5250.528226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5250.531586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.548627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5250.566368] Lustre: Skipped 1 previous similar message [ 5250.576605] Lustre: Skipped 2 previous similar messages [ 5255.649402] LustreError: 175958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.658859] LustreError: 175958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5255.741256] Lustre: server umount lustre-MDT0000 complete [ 5259.472916] LustreError: 175938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784763 with bad export cookie 2360256232285825875 [ 5259.479747] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5259.488042] LustreError: 175938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5259.741889] Lustre: server umount lustre-MDT0001 complete [ 5271.531992] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5274.154893] Key type lgssc unregistered [ 5274.484223] LNet: 179494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5274.499047] LNetError: 179494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5274.527643] LNet: Removed LNI 192.168.203.129@tcp [ 5275.404066] Key type .llcrypt unregistered [ 5275.409987] Key type ._llcrypt unregistered [ 5302.031577] Key type ._llcrypt registered [ 5302.039958] Key type .llcrypt registered [ 5302.167547] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5303.416485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5303.451272] alg: No test for adler32 (adler32-zlib) [ 5304.429458] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5304.627778] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5306.327394] Key type lgssc registered [ 5307.446228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5320.276191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5320.307382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.613076] Lustre: *** cfs_fail_loc=135, val=0*** [ 5321.669135] LustreError: 181020:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5321.675054] LustreError: 181020:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -2 [ 5321.682462] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5321.682462] [ 5321.690034] LustreError: MGC192.168.203.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5321.703688] LustreError: 180981:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5321.709102] LustreError: 180981:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5321.713200] LustreError: 180981:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5321.716749] LustreError: 180981:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5321.774977] Lustre: server umount lustre-MDT0000 complete [ 5321.776758] LustreError: 180981:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5331.049052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.597796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.664399] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5341.155705] LustreError: 181393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.182418] LustreError: 181393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5343.125208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.389288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5347.285472] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5355.619360] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5362.741178] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5367.265898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.272854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.285594] Lustre: Skipped 1 previous similar message [ 5367.303420] Lustre: Skipped 1 previous similar message [ 5368.353770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.432810] Lustre: server umount lustre-MDT0000 complete [ 5373.411667] LustreError: 181393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.434405] LustreError: 181393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5374.975774] LustreError: 181372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784878 with bad export cookie 6092694209208138511 [ 5375.002868] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.543907] Lustre: server umount lustre-MDT0001 complete [ 5386.928722] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5390.221308] Key type lgssc unregistered [ 5390.606739] LNet: 183715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5390.618099] LNetError: 183715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5390.642798] LNet: Removed LNI 192.168.203.129@tcp [ 5391.669675] Key type .llcrypt unregistered [ 5391.680147] Key type ._llcrypt unregistered [ 5421.123976] Key type ._llcrypt registered [ 5421.128159] Key type .llcrypt registered [ 5421.222658] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5421.524436] Key type .llcrypt unregistered [ 5421.530949] Key type ._llcrypt unregistered [ 5438.446354] Key type ._llcrypt registered [ 5438.451910] Key type .llcrypt registered [ 5438.557488] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5439.545450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5439.620532] alg: No test for adler32 (adler32-zlib) [ 5440.682991] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5440.936158] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5442.647227] Key type lgssc registered [ 5443.711985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5450.705667] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5450.731796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5466.207359] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5471.327308] LustreError: 185851:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 5497.952399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5504.596154] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5512.994808] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5519.387388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.875903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.932280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5524.038420] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5524.963699] LustreError: 185877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5524.996304] LustreError: 185877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5530.085082] LustreError: 185876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5532.095650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5532.130693] LustreError: 186889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5532.162132] LustreError: 186889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5532.604557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5537.193595] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5544.825715] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5550.577511] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5553.778710] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5568.491972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.499929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.505518] Lustre: Skipped 1 previous similar message [ 5568.516624] Lustre: Skipped 1 previous similar message [ 5572.230696] Lustre: server umount lustre-OST0000 complete [ 5578.720335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5578.724450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.733129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.741839] Lustre: Skipped 1 previous similar message [ 5582.640502] Lustre: server umount lustre-MDT0000 complete [ 5583.841139] LustreError: 188482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.864019] LustreError: 188482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5586.424650] LustreError: 187630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785090 with bad export cookie 2087734673739043698 [ 5586.434079] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5586.437651] LustreError: 187630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5586.867307] Lustre: server umount lustre-MDT0001 complete [ 5596.823292] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5599.484871] Key type lgssc unregistered [ 5599.847541] LNet: 189869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.860725] LNetError: 189869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5600.936465] LNet: Removed LNI 192.168.203.129@tcp [ 5601.830580] Key type .llcrypt unregistered [ 5601.836238] Key type ._llcrypt unregistered [ 5627.682916] Key type ._llcrypt registered [ 5627.693570] Key type .llcrypt registered [ 5627.848099] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5628.897955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5629.086951] alg: No test for adler32 (adler32-zlib) [ 5630.066259] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5630.292120] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5631.988400] Key type lgssc registered [ 5633.455201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5642.622209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5642.646336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.214873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.257482] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5654.507625] LustreError: 191283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.547829] LustreError: 191283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5659.076748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.486527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5664.401105] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5672.949994] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5679.625908] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5687.865038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.202737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.202989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5695.386249] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5704.382297] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5709.289856] Lustre: Failing over lustre-MDT0000 [ 5709.480986] Lustre: server umount lustre-MDT0000 complete [ 5709.801393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.815750] Lustre: Skipped 2 previous similar messages [ 5709.823872] LustreError: 193172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5709.843823] LustreError: 193172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5711.290439] LustreError: 191280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.928282] LustreError: 192013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.957016] LustreError: 192013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5720.033623] LustreError: 191283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.067043] LustreError: 191283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5729.274434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.401442] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5729.643734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.679807] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5731.723123] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5734.433607] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5734.895020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5734.925047] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5734.976912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5738.609726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5746.336581] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5759.977837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5759.983384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.000254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5760.995174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5761.002594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.020395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.557158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.564155] Lustre: Skipped 1 previous similar message [ 5765.461895] Lustre: server umount lustre-OST0000 complete [ 5770.720386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5770.732651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.767060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.203455] Lustre: server umount lustre-MDT0000 complete [ 5778.916611] LustreError: 191279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.933419] LustreError: 191279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5780.231651] LustreError: 193916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785284 with bad export cookie 8850523256257889773 [ 5780.235372] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5780.245241] LustreError: 193916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5780.542034] Lustre: server umount lustre-MDT0001 complete [ 5793.229889] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5796.427514] Key type lgssc unregistered [ 5796.787969] LNet: 196210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5796.802889] LNetError: 196210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5796.821901] LNet: Removed LNI 192.168.203.129@tcp [ 5797.649312] Key type .llcrypt unregistered [ 5797.652375] Key type ._llcrypt unregistered [ 5822.792455] Key type ._llcrypt registered [ 5822.795150] Key type .llcrypt registered [ 5822.904675] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5824.273578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5824.366745] alg: No test for adler32 (adler32-zlib) [ 5825.389671] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5825.690286] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5827.401190] Key type lgssc registered [ 5828.851687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5837.293327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5837.325400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.853634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.849396] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5849.061553] LustreError: 197623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.091516] LustreError: 197623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5852.498992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.014238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.358707] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5865.194864] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5872.029392] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5879.411771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.796769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.912981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5887.445587] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5896.991118] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5900.060744] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5906.218170] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5920.042466] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5924.834353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.842701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.869771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5927.917446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5927.919991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5927.920102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.925240] Lustre: Skipped 1 previous similar message [ 5930.547586] Lustre: server umount lustre-OST0000 complete [ 5935.074328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5935.090281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.120928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.146436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.152746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.173237] Lustre: Skipped 1 previous similar message [ 5940.978698] Lustre: server umount lustre-MDT0000 complete [ 5943.269406] LustreError: 197623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.290472] LustreError: 197623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5944.925174] LustreError: 197602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785448 with bad export cookie 5187460408611450232 [ 5944.929575] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5944.934943] LustreError: 197602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5945.253979] Lustre: server umount lustre-MDT0001 complete [ 5956.416973] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5959.256978] Key type lgssc unregistered [ 5959.491530] LNet: 201478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5959.504191] LNetError: 201478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5959.536354] LNet: Removed LNI 192.168.203.129@tcp [ 5960.656261] Key type .llcrypt unregistered [ 5960.663217] Key type ._llcrypt unregistered [ 5980.024980] Key type ._llcrypt registered [ 5980.028187] Key type .llcrypt registered [ 5980.137800] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5980.636334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5980.691559] alg: No test for adler32 (adler32-zlib) [ 5981.779571] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 5982.143824] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5983.833359] Key type lgssc registered [ 5985.195520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5994.204232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5994.216748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.693994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.385202] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6005.221381] LustreError: 202744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.253906] LustreError: 202744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6010.336788] LustreError: 202743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.384469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6011.910856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6017.316322] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6026.119651] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6032.642792] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6040.191088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6040.553916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.680577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6047.433657] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6055.942492] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6063.232587] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6075.296523] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6082.341969] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6088.279614] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6097.890931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6097.893279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.896096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6097.918281] Lustre: Skipped 1 previous similar message [ 6103.009871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.035974] Lustre: Skipped 2 previous similar messages [ 6108.138343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6108.150455] Lustre: Skipped 1 previous similar message [ 6112.223721] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6112.464286] Lustre: server umount lustre-OST0000 complete [ 6118.370062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.371917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6118.385369] Lustre: Skipped 1 previous similar message [ 6118.402138] Lustre: Skipped 1 previous similar message [ 6123.490879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.515027] Lustre: Skipped 1 previous similar message [ 6123.754165] Lustre: server umount lustre-MDT0000 complete [ 6127.678774] LustreError: 202726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785631 with bad export cookie 6382795661237593337 [ 6127.679295] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.703496] LustreError: 202726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6127.941198] Lustre: server umount lustre-MDT0001 complete [ 6139.454029] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 6142.844785] Key type lgssc unregistered [ 6143.205371] LNet: 207577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6143.217078] LNetError: 207577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6143.233295] LNet: Removed LNI 192.168.203.129@tcp [ 6143.991386] Key type .llcrypt unregistered [ 6143.995684] Key type ._llcrypt unregistered [ 6170.182925] Key type ._llcrypt registered [ 6170.188669] Key type .llcrypt registered [ 6170.296369] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6171.504587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6171.598088] alg: No test for adler32 (adler32-zlib) [ 6172.653192] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 6172.951620] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6174.599166] Key type lgssc registered [ 6175.633225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6183.047538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6183.060290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.582715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.180496] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6194.146128] LustreError: 208991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.155849] LustreError: 208991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6197.537640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.056671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.232108] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6209.855532] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6216.081872] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6222.577299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.007460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.556918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6229.559541] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6237.793752] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.053456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.063625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.087066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.082574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.088198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.100452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.994546] Lustre: server umount lustre-OST0000 complete [ 6264.800247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6264.810073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.826450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6264.830521] Lustre: Skipped 1 previous similar message [ 6267.873698] Lustre: server umount lustre-MDT0000 complete [ 6270.434974] LustreError: 209010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.449097] LustreError: 209010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6271.414055] LustreError: 208972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785775 with bad export cookie 3401160001309871053 [ 6271.416267] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.431142] LustreError: 208972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6271.781620] Lustre: server umount lustre-MDT0001 complete [ 6283.260434] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 6286.585447] Key type lgssc unregistered [ 6286.941944] LNet: 212680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6286.956945] LNetError: 212680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6286.975708] LNet: Removed LNI 192.168.203.129@tcp [ 6288.149900] Key type .llcrypt unregistered [ 6288.152141] Key type ._llcrypt unregistered [ 6307.292421] Key type ._llcrypt registered [ 6307.294631] Key type .llcrypt registered [ 6307.375744] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6308.683800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6308.783860] alg: No test for adler32 (adler32-zlib) [ 6309.834121] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 6310.065257] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6311.832120] Key type lgssc registered [ 6313.050598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6320.897841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6320.933190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.408241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.437057] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6332.904710] LustreError: 213946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.927209] LustreError: 213946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6335.080084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.466159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.261366] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6347.342463] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6353.004571] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.296961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.802408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.083964] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6367.992392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6373.735431] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6386.663396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.675482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.690148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6388.710103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.714361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6388.714843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6388.747617] Lustre: Skipped 1 previous similar message [ 6392.789461] Lustre: server umount lustre-OST0000 complete [ 6396.898823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6396.914276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.935109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6398.944574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.955789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6398.985394] Lustre: Skipped 1 previous similar message [ 6404.070962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.081881] Lustre: Skipped 1 previous similar message [ 6411.231169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6411.524866] Lustre: server umount lustre-MDT0000 complete [ 6413.280764] LustreError: 213947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.300166] LustreError: 213947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6415.791026] LustreError: 213926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785919 with bad export cookie 4766195481236738793 [ 6415.792108] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.797785] LustreError: 213926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6416.228973] Lustre: server umount lustre-MDT0001 complete [ 6426.428317] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 6429.152284] Key type lgssc unregistered [ 6429.489873] LNet: 217639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6429.497956] LNetError: 217639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6429.517582] LNet: Removed LNI 192.168.203.129@tcp [ 6430.440862] Key type .llcrypt unregistered [ 6430.445434] Key type ._llcrypt unregistered [ 6448.690785] Key type ._llcrypt registered [ 6448.693771] Key type .llcrypt registered [ 6448.816544] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6449.878325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6450.076343] alg: No test for adler32 (adler32-zlib) [ 6451.232694] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 6451.526260] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6453.167242] Key type lgssc registered [ 6454.378624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6462.454525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6462.486348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.940198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.480479] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6473.191860] LustreError: 218910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.207982] LustreError: 218910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6476.865311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.372227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6482.378804] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6490.600272] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6496.633875] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6503.423421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.848297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.983688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6510.267667] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6518.212763] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6527.422839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6527.561920] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6527.768649] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6534.527087] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6541.670066] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6543.228357] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6554.769790] Lustre: Permanently deactivating lustre-OST0001 [ 6554.782901] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6562.388104] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6568.849444] Lustre: Permanently reactivating lustre-OST0001 [ 6568.873721] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6568.879494] Lustre: Skipped 2 previous similar messages [ 6578.816180] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.830910] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6578.847352] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6578.862381] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6578.920443] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6581.741377] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.751890] Lustre: Skipped 1 previous similar message [ 6581.763334] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6581.769713] Lustre: Skipped 1 previous similar message [ 6581.917162] Lustre: server umount lustre-OST0001 complete [ 6585.825410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.837975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.862955] Lustre: Skipped 1 previous similar message [ 6585.868172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.871089] LustreError: 220732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.873798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.911036] LustreError: 220732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6586.949262] Lustre: Skipped 1 previous similar message [ 6591.650102] Lustre: server umount lustre-OST0000 complete [ 6595.041380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6595.049170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.086930] Lustre: Skipped 1 previous similar message [ 6595.108618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.208215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.213334] Lustre: Skipped 3 previous similar messages [ 6609.375227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6611.427147] LustreError: 218905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.447119] LustreError: 218905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6611.595522] Lustre: server umount lustre-MDT0000 complete [ 6615.271642] LustreError: 221748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786119 with bad export cookie 16593636759384018341 [ 6615.277057] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.289117] LustreError: 221748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6615.558678] Lustre: server umount lustre-MDT0001 complete [ 6625.630120] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 6628.577855] Key type lgssc unregistered [ 6628.910636] LNet: 223960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6628.922593] LNetError: 223960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6628.944359] LNet: Removed LNI 192.168.203.129@tcp [ 6629.831799] Key type .llcrypt unregistered [ 6629.838086] Key type ._llcrypt unregistered [ 6666.441252] Key type ._llcrypt registered [ 6666.443621] Key type .llcrypt registered [ 6666.558894] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6667.654774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6667.679783] alg: No test for adler32 (adler32-zlib) [ 6668.707395] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 6668.945287] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6670.647459] Key type lgssc registered [ 6671.734501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6678.664569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6678.685244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.951567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6679.985131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6680.144671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.750091] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6693.376366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.472487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6693.511799] Lustre: 226166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6693.819930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6698.065392] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6706.641873] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6712.568834] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.839374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6719.107733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6719.432477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6725.379905] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6728.795493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6733.031666] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6737.537834] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6744.541575] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6753.742553] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6763.910194] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6775.010946] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6775.016327] Lustre: Skipped 1 previous similar message [ 6798.732626] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6798.738885] Lustre: Skipped 2 previous similar messages [ 6833.942547] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6833.948245] Lustre: Skipped 3 previous similar messages [ 6907.826616] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6907.854994] Lustre: Skipped 8 previous similar messages [ 6918.116031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6918.119871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.121096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6918.141630] Lustre: Skipped 1 previous similar message [ 6919.964494] Lustre: server umount lustre-OST0000 complete [ 6923.238193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.259668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6923.262690] Lustre: Skipped 2 previous similar messages [ 6928.355951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6928.378876] Lustre: Skipped 1 previous similar message [ 6929.408181] Lustre: server umount lustre-MDT0000 complete [ 6933.327456] LustreError: 226156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786437 with bad export cookie 18009575950402738390 [ 6933.334768] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.341281] LustreError: 226156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6933.710702] Lustre: server umount lustre-MDT0001 complete [ 6943.515470] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 6945.998775] Key type lgssc unregistered [ 6946.301885] LNet: 230954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6946.312487] LNetError: 230954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6946.341234] LNet: Removed LNI 192.168.203.129@tcp [ 6947.147114] Key type .llcrypt unregistered [ 6947.150763] Key type ._llcrypt unregistered [ 6966.750293] Key type ._llcrypt registered [ 6966.753043] Key type .llcrypt registered [ 6966.829800] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6967.581533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6967.663179] alg: No test for adler32 (adler32-zlib) [ 6968.733412] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 6968.944028] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6970.607174] Key type lgssc registered [ 6971.563937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6978.498072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6978.531213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.900645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.143442] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6989.291262] LustreError: 232347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.311369] LustreError: 232347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6991.744874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.070250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6996.243897] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7003.141569] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7008.395378] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7014.231689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7014.512767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.027957] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7024.751563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7026.559624] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.529769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.539164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.551908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7043.886379] Lustre: server umount lustre-OST0000 complete [ 7048.674703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7048.683870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.704201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.723784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.728639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.840304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7055.843096] Lustre: Skipped 2 previous similar messages [ 7057.971660] Lustre: server umount lustre-MDT0000 complete [ 7060.963518] LustreError: 233400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.977450] LustreError: 233400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7061.273124] LustreError: 232327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786565 with bad export cookie 15933332521074153764 [ 7061.281083] LustreError: 232327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7061.281456] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.625702] Lustre: server umount lustre-MDT0001 complete [ 7071.140091] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7073.655514] Key type lgssc unregistered [ 7073.964398] LNet: 236038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7073.990035] LNetError: 236038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7074.010266] LNet: Removed LNI 192.168.203.129@tcp [ 7074.931768] Key type .llcrypt unregistered [ 7074.934316] Key type ._llcrypt unregistered [ 7097.702652] Key type ._llcrypt registered [ 7097.705844] Key type .llcrypt registered [ 7097.822130] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7098.626143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7098.715724] alg: No test for adler32 (adler32-zlib) [ 7099.754688] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 7099.942684] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7101.583512] Key type lgssc registered [ 7102.489167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7375.985677] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7379.308327] Key type lgssc unregistered [ 7379.691605] LNet: 238065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7379.705585] LNetError: 238065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7379.723628] LNet: Removed LNI 192.168.203.129@tcp [ 7380.554578] Key type .llcrypt unregistered [ 7380.558341] Key type ._llcrypt unregistered [ 7406.257874] Key type ._llcrypt registered [ 7406.260323] Key type .llcrypt registered [ 7406.337319] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 7422.475137] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7423.555268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7423.598396] alg: No test for adler32 (adler32-zlib) [ 7424.771686] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 7425.065968] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7426.775189] Key type lgssc registered [ 7428.071517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7462.150251] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7213 sec ========== 19:29:24 (1787786964) [ 7464.291524] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:29:25 (1787786965) === [ 7467.974427] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:29:29 (1787786969) === [ 7510.903626] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7513.587152] Key type lgssc unregistered [ 7513.852864] LNet: 243310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7513.864360] LNetError: 243310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7513.879276] LNet: Removed LNI 192.168.203.129@tcp [ 7514.627688] Key type .llcrypt unregistered [ 7514.632172] Key type ._llcrypt unregistered