[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 434559660 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003204] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010178] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021283] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025845] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027081] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029054] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032006] *** VALIDATE proc *** [ 0.033230] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038055] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042467] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.044185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045720] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046014] ... version: 2 [ 0.047021] ... bit width: 48 [ 0.048007] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050007] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052007] ... event mask: 000000070000000f [ 0.053339] rcu: Hierarchical SRCU implementation. [ 0.055659] smp: Bringing up secondary CPUs ... [ 0.056552] x86: Booting SMP configuration: [ 0.057014] .... node #0, CPUs: #1 #2 #3 [ 0.060260] smp: Brought up 1 node, 4 CPUs [ 0.062009] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.233640] node 0 deferred pages initialised in 169ms [ 0.236174] devtmpfs: initialized [ 0.237245] x86/mm: Memory block size: 128MB [ 0.239828] gcov: version magic: 0x41383552 [ 0.241414] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243283] pinctrl core: initialized pinctrl subsystem [ 0.244192] [ 0.244639] ************************************************************* [ 0.245008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246008] ** ** [ 0.247006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248007] ** ** [ 0.249008] ** This means that this kernel is built to expose internal ** [ 0.250026] ** IOMMU data structures, which may compromise security on ** [ 0.251008] ** your system. ** [ 0.252009] ** ** [ 0.253008] ** If you see this message and you are not debugging the ** [ 0.254009] ** kernel, report this immediately to your vendor! ** [ 0.255008] ** ** [ 0.256008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257006] ************************************************************* [ 0.258662] NET: Registered protocol family 16 [ 0.259458] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.262555] cpuidle: using governor menu [ 0.264512] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.266496] PCI: Using configuration type 1 for base access [ 0.268112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.274075] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.276024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.279028] cryptd: max_cpu_qlen set to 1000 [ 0.281105] ACPI: Added _OSI(Module Device) [ 0.282006] ACPI: Added _OSI(Processor Device) [ 0.283005] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.284009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.286935] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.291395] ACPI: Interpreter enabled [ 0.293043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.293979] ACPI: Using IOAPIC for interrupt routing [ 0.295064] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.297319] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304979] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.307008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.310064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.313041] acpiphp: Slot [2] registered [ 0.313982] acpiphp: Slot [5] registered [ 0.315069] acpiphp: Slot [6] registered [ 0.315992] acpiphp: Slot [7] registered [ 0.316057] acpiphp: Slot [8] registered [ 0.317028] acpiphp: Slot [9] registered [ 0.319082] acpiphp: Slot [10] registered [ 0.320008] acpiphp: Slot [3] registered [ 0.320870] acpiphp: Slot [4] registered [ 0.322072] acpiphp: Slot [11] registered [ 0.323053] acpiphp: Slot [12] registered [ 0.323952] acpiphp: Slot [13] registered [ 0.324069] acpiphp: Slot [14] registered [ 0.326078] acpiphp: Slot [15] registered [ 0.327059] acpiphp: Slot [16] registered [ 0.329180] acpiphp: Slot [17] registered [ 0.331078] acpiphp: Slot [18] registered [ 0.332072] acpiphp: Slot [19] registered [ 0.334066] acpiphp: Slot [20] registered [ 0.335063] acpiphp: Slot [21] registered [ 0.336088] acpiphp: Slot [22] registered [ 0.338081] acpiphp: Slot [23] registered [ 0.339094] acpiphp: Slot [24] registered [ 0.340047] acpiphp: Slot [25] registered [ 0.342073] acpiphp: Slot [26] registered [ 0.343051] acpiphp: Slot [27] registered [ 0.344012] acpiphp: Slot [28] registered [ 0.344934] acpiphp: Slot [29] registered [ 0.346056] acpiphp: Slot [30] registered [ 0.346982] acpiphp: Slot [31] registered [ 0.348063] PCI host bridge to bus 0000:00 [ 0.348957] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.353016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.356013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.359031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.362015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.364183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.368163] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371461] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.383947] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.391021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.393013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.396011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.399754] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.401599] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.403019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404949] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408929] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.417962] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423994] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.426945] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430725] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454814] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.467015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.491782] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.528512] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532900] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.537010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.552013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563012] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.569011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.601473] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.608009] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.613010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.630012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.640506] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.643402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.646324] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.649349] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.652210] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.656210] iommu: Default domain type: Passthrough [ 0.659608] SCSI subsystem initialized [ 0.661148] ACPI: bus type USB registered [ 0.663076] usbcore: registered new interface driver usbfs [ 0.665047] usbcore: registered new interface driver hub [ 0.666073] usbcore: registered new device driver usb [ 0.668119] pps_core: LinuxPPS API ver. 1 registered [ 0.669005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.671052] PTP clock support registered [ 0.673125] EDAC MC: Ver: 3.0.0 [ 0.674421] PCI: Using ACPI for IRQ routing [ 0.677011] NetLabel: Initializing [ 0.679020] NetLabel: domain hash size = 128 [ 0.681011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.683081] NetLabel: unlabeled traffic allowed by default [ 0.686190] vgaarb: loaded [ 0.688275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.690012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.697005] clocksource: Switched to clocksource kvm-clock [ 0.799546] VFS: Disk quotas dquot_6.6.0 [ 0.800613] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802843] *** VALIDATE ramfs *** [ 0.804161] *** VALIDATE hugetlbfs *** [ 0.805624] pnp: PnP ACPI init [ 0.808445] pnp: PnP ACPI: found 6 devices [ 0.821531] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.824701] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.826854] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.828778] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.830966] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.833304] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.835838] NET: Registered protocol family 2 [ 0.838102] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.842745] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.846120] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.851500] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.855108] TCP: Hash tables configured (established 65536 bind 65536) [ 0.858349] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.861470] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.867968] NET: Registered protocol family 1 [ 0.870202] RPC: Registered named UNIX socket transport module. [ 0.873219] RPC: Registered udp transport module. [ 0.876813] RPC: Registered tcp transport module. [ 0.879053] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.881288] NET: Registered protocol family 44 [ 0.882275] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.883835] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885131] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.886645] PCI: CLS 0 bytes, default 64 [ 0.888438] Unpacking initramfs... [ 2.240841] debug: unmapping init [mem 0xffff9cbebcc54000-0xffff9cbebffbffff] [ 2.244338] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.246350] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.248957] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.754086] Initialise system trusted keyrings [ 2.755398] Key type blacklist registered [ 2.756891] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.765401] zbud: loaded [ 2.767984] *** VALIDATE nfs *** [ 2.770428] *** VALIDATE nfs4 *** [ 2.771497] pstore: using deflate compression [ 2.774611] Platform Keyring initialized [ 2.871425] NET: Registered protocol family 38 [ 2.873098] Key type asymmetric registered [ 2.874412] Asymmetric key parser 'x509' registered [ 2.875850] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.878299] io scheduler mq-deadline registered [ 2.879639] io scheduler kyber registered [ 2.881609] io scheduler bfq registered [ 2.883433] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.886325] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.888548] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.892272] ACPI: Power Button [PWRF] [ 2.897475] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.903426] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.917730] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.929236] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.945412] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.975691] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.007053] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.012783] Non-volatile memory driver v1.3 [ 3.014575] Linux agpgart interface v0.103 [ 3.043858] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.046575] vda: detected capacity change from 0 to 74715136 [ 3.058854] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.061710] vdb: detected capacity change from 0 to 1073741824 [ 3.076490] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.079980] vdc: detected capacity change from 0 to 2621440000 [ 3.096927] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.100502] vdd: detected capacity change from 0 to 2621440000 [ 3.116925] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.120490] vde: detected capacity change from 0 to 4294967296 [ 3.136195] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.141743] vdf: detected capacity change from 0 to 4294967296 [ 3.150171] libphy: Fixed MDIO Bus: probed [ 3.155770] usbcore: registered new interface driver usbserial_generic [ 3.161182] usbserial: USB Serial support registered for generic [ 3.165541] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.171812] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.173652] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.175939] mousedev: PS/2 mouse device common for all mice [ 3.179598] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.182481] rtc_cmos 00:05: RTC can wake from S4 [ 3.188140] rtc_cmos 00:05: registered as rtc0 [ 3.189855] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.192087] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.196017] intel_pstate: CPU model not supported [ 3.202407] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.207698] hid: raw HID events driver (C) Jiri Kosina [ 3.210117] usbcore: registered new interface driver usbhid [ 3.214424] usbhid: USB HID core driver [ 3.216809] drop_monitor: Initializing network drop monitor service [ 3.223285] Initializing XFRM netlink socket [ 3.226623] NET: Registered protocol family 10 [ 3.230029] Segment Routing with IPv6 [ 3.231846] NET: Registered protocol family 17 [ 3.233614] mpls_gso: MPLS GSO support [ 3.240393] RAS: Correctable Errors collector initialized. [ 3.243128] AVX version of gcm_enc/dec engaged. [ 3.244860] AES CTR mode by8 optimization enabled [ 3.354793] sched_clock: Marking stable (3354716780, 0)->(4304039497, -949322717) [ 3.360359] registered taskstats version 1 [ 3.362591] Loading compiled-in X.509 certificates [ 3.365229] zswap: loaded using pool lzo/zbud [ 3.402727] Key type big_key registered [ 3.428716] Key type encrypted registered [ 3.430376] ima: No TPM chip found, activating TPM-bypass! [ 3.432388] ima: Allocated hash algorithm: sha1 [ 3.434188] ima: No architecture policies found [ 3.436100] evm: Initialising EVM extended attributes: [ 3.437880] evm: security.selinux [ 3.439300] evm: security.ima [ 3.440550] evm: security.capability [ 3.441828] evm: HMAC attrs: 0x1 [ 3.444720] rtc_cmos 00:05: setting system clock to 2026-08-12 13:33:57 UTC (1786541637) [ 3.453480] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.457349] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 3.466291] Write protecting the kernel read-only data: 28672k [ 3.470572] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.473879] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 3.510431] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.516903] systemd[1]: Detected virtualization kvm. [ 3.518770] systemd[1]: Detected architecture x86-64. [ 3.520842] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.550508] systemd[1]: No hostname configured. [ 3.552277] systemd[1]: Set hostname to . [ 3.553965] random: systemd: uninitialized urandom read (16 bytes read) [ 3.555863] systemd[1]: Initializing machine ID from random generator. [ 3.622260] random: ln: uninitialized urandom read (6 bytes read) [ 3.743698] random: systemd: uninitialized urandom read (16 bytes read) [ 3.746453] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.754820] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.762624] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.598792] device-mapper: uevent: version 1.0.3 [ 4.600986] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK [ 5.522775] random: fast init done ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.653682] virtio_net virtio0 ens2: renamed from eth0 [ 5.662782] scsi host0: ata_piix [ 5.676957] scsi host1: ata_piix [ 5.682872] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.686030] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.952760] random: crng init done [ 10.954095] random: 7 urandom warning(s) missed due to ratelimiting [ 10.990820] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.440493] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.050223] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.430868] SELinux: Disabled at runtime. [ 14.498737] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.509415] systemd[1]: Detected virtualization kvm. [ 14.511455] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.310570] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.313745] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.326534] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.335533] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.344910] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.356830] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.401147] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 15.484665] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.202216] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.783326] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.803122] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.129069] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.156948] EDAC sbridge: Ver: 1.1.2 [ 19.604252] Key type dns_resolver registered [ 19.999620] NFS: Registering the id_resolver key type [ 20.002461] Key type id_resolver registered [ 20.004194] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 42.852682] libcfs: loading out-of-tree module taints kernel. [ 42.867287] Key type ._llcrypt registered [ 42.868244] Key type .llcrypt registered [ 42.900181] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 49.452040] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 49.895043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.900460] alg: No test for adler32 (adler32-zlib) [ 50.893347] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 51.245077] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 52.863169] Key type lgssc registered [ 53.417678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.650688] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 72.984540] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 77.051290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 77.060612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.147478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 78.159198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 78.197715] Lustre: lustre-MDT0000: new disk, initializing [ 78.220719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 78.229728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 79.515695] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.741656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 84.783923] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 84.798783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 84.801079] Lustre: Skipped 1 previous similar message [ 84.842625] Lustre: lustre-MDT0001: new disk, initializing [ 84.862639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 84.873222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 84.875951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 86.216043] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.468266] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 91.723680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 91.817319] Lustre: lustre-OST0000: new disk, initializing [ 91.819290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 91.821665] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 91.837772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.813761] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.019736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 95.022548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 95.043282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 99.285447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 99.334885] Lustre: lustre-OST0001: new disk, initializing [ 99.336390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 99.338562] Lustre: 9490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 99.356995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.451937] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.699371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 102.702736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 102.713944] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 107.277500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 112.957876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 114.978074] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 116.378495] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 117.751038] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 118.547983] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 119.301988] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 119.796872] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 12 09:35:54 EDT 2026 [ 125.032158] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 125.548635] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 138.719686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 138.720112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.723168] Lustre: Skipped 2 previous similar messages [ 138.726210] Lustre: Skipped 1 previous similar message [ 143.384251] Lustre: server umount lustre-MDT0000 complete [ 144.486269] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541778 with bad export cookie 1653572108048070988 [ 144.487534] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 144.489766] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 148.959582] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 148.959679] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.963207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 148.963212] Lustre: Skipped 2 previous similar messages [ 148.968398] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 148.972664] Lustre: Skipped 2 previous similar messages [ 149.983581] LustreError: 11149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 149.983767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.990587] LustreError: 11149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 149.997751] Lustre: Skipped 2 previous similar messages [ 150.631342] Lustre: server umount lustre-MDT0001 complete [ 166.367131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 166.402137] Lustre: server umount lustre-OST0000 complete [ 173.699969] Lustre: server umount lustre-OST0001 complete [ 175.587509] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 178.337725] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 189.003258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.080723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.090023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.123972] Lustre: lustre-MDT0000: new disk, initializing [ 189.143581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.149577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.392784] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 194.404929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.446157] Lustre: 15032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.545045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.785278] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 199.205546] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 200.951742] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 202.974452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.066908] Lustre: lustre-OST0000: new disk, initializing [ 203.068168] Lustre: Skipped 1 previous similar message [ 203.069645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.071116] Lustre: Skipped 2 previous similar messages [ 203.073104] Lustre: 16998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.100518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.102267] Lustre: Skipped 1 previous similar message [ 204.600847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.605583] Lustre: Skipped 1 previous similar message [ 204.608912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.646571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.889156] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 208.457677] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 209.821851] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 209.906591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 211.142071] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 211.214525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 215.011183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 215.011339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.021870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 215.023866] Lustre: Skipped 1 previous similar message [ 220.130548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 220.132377] Lustre: Skipped 1 previous similar message [ 226.271190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 226.328209] Lustre: server umount lustre-OST0000 complete [ 230.369034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 230.369244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.370410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 230.370415] Lustre: Skipped 2 previous similar messages [ 230.371553] LustreError: Skipped 1 previous similar message [ 230.376250] Lustre: Skipped 2 previous similar messages [ 233.818951] Lustre: server umount lustre-MDT0000 complete [ 235.100178] LustreError: 15827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541869 with bad export cookie 1653572108048072521 [ 235.103570] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 235.104881] LustreError: 15827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 235.226763] Lustre: server umount lustre-MDT0001 complete [ 236.398836] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:37:50 (1786541870) [ 238.657285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.808803] LustreError: 19258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 238.832100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.202576] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 243.242936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.833416] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 247.202820] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 249.077061] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 251.382695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.519379] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 255.881094] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 269.282559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 269.283309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.284133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 269.284137] Lustre: Skipped 1 previous similar message [ 269.285426] LustreError: Skipped 1 previous similar message [ 269.289159] Lustre: Skipped 1 previous similar message [ 275.024442] Lustre: server umount lustre-OST0000 complete [ 279.520673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 279.520892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.529675] Lustre: Skipped 1 previous similar message [ 282.326167] Lustre: server umount lustre-MDT0000 complete [ 283.472534] LustreError: 19239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541917 with bad export cookie 1653572108048073403 [ 283.475129] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.478080] LustreError: 19239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 283.591155] Lustre: server umount lustre-MDT0001 complete [ 286.964319] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 287.925281] Key type lgssc unregistered [ 288.050313] LNet: 22812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.052441] LNetError: 22812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.062280] LNet: Removed LNI 192.168.204.131@tcp [ 288.381101] Key type .llcrypt unregistered [ 288.382189] Key type ._llcrypt unregistered [ 296.768108] Key type ._llcrypt registered [ 296.769252] Key type .llcrypt registered [ 296.804525] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 297.151595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 297.182908] alg: No test for adler32 (adler32-zlib) [ 298.041883] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 298.125396] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 299.703157] Key type lgssc registered [ 300.080124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 302.704115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 302.709477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.852500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 305.099879] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 307.800891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.925949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 309.279419] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 311.588167] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 313.298080] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 315.407183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 315.527831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 317.457973] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 319.781732] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 328.866992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 334.305459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.306269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 334.306808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 334.310261] Lustre: Skipped 1 previous similar message [ 339.425507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 339.428591] Lustre: Skipped 2 previous similar messages [ 340.302454] Lustre: server umount lustre-OST0000 complete [ 344.031835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 344.034361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.040549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.608389] Lustre: server umount lustre-MDT0000 complete [ 348.916166] LustreError: 24182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541982 with bad export cookie 8353612395802254398 [ 348.919474] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.920916] LustreError: 24182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.041379] Lustre: server umount lustre-MDT0001 complete [ 352.620867] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 353.717677] Key type lgssc unregistered [ 353.853481] LNet: 28200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.857358] LNetError: 28200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 353.866306] LNet: Removed LNI 192.168.204.131@tcp [ 354.205485] Key type .llcrypt unregistered [ 354.207287] Key type ._llcrypt unregistered [ 363.055349] Key type ._llcrypt registered [ 363.056697] Key type .llcrypt registered [ 363.091148] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 363.462304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 363.497270] alg: No test for adler32 (adler32-zlib) [ 364.349435] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 364.428674] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 365.999161] Key type lgssc registered [ 366.362930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 368.937149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 368.943877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.107436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.395014] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 374.018404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.123517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 375.397072] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 377.876988] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 379.752800] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 384.086784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 384.191165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.010533] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 388.274523] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 395.494944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 400.864204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 400.867063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.871741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 400.874822] Lustre: Skipped 1 previous similar message [ 402.320136] Lustre: server umount lustre-OST0000 complete [ 404.959924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 404.962643] LustreError: Skipped 1 previous similar message [ 404.964166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.968595] Lustre: Skipped 1 previous similar message [ 404.970279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.584933] Lustre: server umount lustre-MDT0000 complete [ 405.984051] LustreError: 29589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.984432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.988305] LustreError: 29589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 406.693279] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542040 with bad export cookie 8217766376470605413 [ 406.696166] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.696821] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 406.804684] Lustre: server umount lustre-MDT0001 complete [ 410.213673] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 411.280607] Key type lgssc unregistered [ 411.403675] LNet: 33486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.407178] LNetError: 33486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.415312] LNet: Removed LNI 192.168.204.131@tcp [ 411.729789] Key type .llcrypt unregistered [ 411.731348] Key type ._llcrypt unregistered [ 420.139661] Key type ._llcrypt registered [ 420.140912] Key type .llcrypt registered [ 420.172724] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 420.466901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 420.474973] alg: No test for adler32 (adler32-zlib) [ 421.314058] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 421.396123] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 422.967155] Key type lgssc registered [ 423.273673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 425.730190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 425.736344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.872508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.169664] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 430.829225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.946700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.239916] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 434.604944] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.318089] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 438.336646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.435394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.176643] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 442.303644] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 451.746599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 456.159777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.162233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.167121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.185514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.186521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.186562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.452042] Lustre: server umount lustre-OST0000 complete [ 466.912093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.914370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.918127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.921687] Lustre: Skipped 1 previous similar message [ 468.837549] Lustre: server umount lustre-MDT0000 complete [ 470.076110] LustreError: 37675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542104 with bad export cookie 12548936385892983667 [ 470.079790] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.080531] LustreError: 37675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 470.207041] Lustre: server umount lustre-MDT0001 complete [ 473.834130] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 474.841956] Key type lgssc unregistered [ 474.964359] LNet: 38428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.966661] LNetError: 38428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.976335] LNet: Removed LNI 192.168.204.131@tcp [ 475.265547] Key type .llcrypt unregistered [ 475.267214] Key type ._llcrypt unregistered [ 483.815793] Key type ._llcrypt registered [ 483.816842] Key type .llcrypt registered [ 483.855501] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 484.188894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 484.248647] alg: No test for adler32 (adler32-zlib) [ 485.109225] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 485.207193] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 486.791192] Key type lgssc registered [ 487.166159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 489.828694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 489.835450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.975860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.360173] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 495.333338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.478989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 496.831230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 499.391774] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 501.285712] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 503.539407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 503.647322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.592342] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 508.071359] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.962970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 521.184568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.189027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.193280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.210149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.210254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.210700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.219088] Lustre: Skipped 1 previous similar message [ 523.929388] Lustre: lustre-OST0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 527.121820] Lustre: server umount lustre-OST0000 complete [ 539.869923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.874512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.876850] Lustre: Skipped 1 previous similar message [ 545.247911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.249887] Lustre: Skipped 1 previous similar message [ 546.065897] Lustre: server umount lustre-MDT0000 complete [ 547.232385] LustreError: 39799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542181 with bad export cookie 14116580405740917099 [ 547.234576] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.236823] LustreError: 39799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.384691] Lustre: server umount lustre-MDT0001 complete [ 556.816629] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 557.866251] Key type lgssc unregistered [ 557.990373] LNet: 43544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.992710] LNetError: 43544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.001295] LNet: Removed LNI 192.168.204.131@tcp [ 558.283823] Key type .llcrypt unregistered [ 558.284935] Key type ._llcrypt unregistered [ 566.340867] Key type ._llcrypt registered [ 566.341844] Key type .llcrypt registered [ 566.376423] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 566.711392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 566.742338] alg: No test for adler32 (adler32-zlib) [ 567.583242] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 567.656878] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 569.231135] Key type lgssc registered [ 569.554526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 571.883279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 571.888779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.014078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.174905] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 577.015731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.140915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 578.501274] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 580.973850] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.877401] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.200829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 585.319935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.333839] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 589.855457] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 598.630682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 602.592775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.595387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.599901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 604.129261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 604.129974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.337322] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 605.339762] Lustre: Skipped 2 previous similar messages [ 608.737795] Lustre: server umount lustre-MDT0000 complete [ 609.248800] LustreError: 44932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.254319] LustreError: 44932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 609.981038] LustreError: 46772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542244 with bad export cookie 230160215649408035 [ 609.984525] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.988083] LustreError: 46772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 610.450582] LustreError: 44928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.456967] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 614.367938] LustreError: 44926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.368259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.379221] Lustre: Skipped 1 previous similar message [ 615.570441] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 615.573031] Lustre: Skipped 1 previous similar message [ 616.174101] Lustre: server umount lustre-MDT0001 complete [ 633.939547] Lustre: server umount lustre-OST0000 complete [ 639.455619] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 640.535465] Key type lgssc unregistered [ 640.684439] LNet: 48810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.688694] LNetError: 48810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.701398] LNet: Removed LNI 192.168.204.131@tcp [ 641.010874] Key type .llcrypt unregistered [ 641.012393] Key type ._llcrypt unregistered [ 650.840125] Key type ._llcrypt registered [ 650.841271] Key type .llcrypt registered [ 650.875564] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 651.201602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 651.263075] alg: No test for adler32 (adler32-zlib) [ 652.124227] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 652.209117] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 653.783190] Key type lgssc registered [ 654.178962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 656.805643] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.811311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.223361] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 677.343371] LustreError: 50175:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 703.967237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.844428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 708.245291] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 873.600642] Lustre: server umount lustre-OST0000 complete [ 879.324662] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 880.310860] Key type lgssc unregistered [ 880.429299] LNet: 51855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.432770] LNetError: 51855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.446342] LNet: Removed LNI 192.168.204.131@tcp [ 880.758629] Key type .llcrypt unregistered [ 880.759805] Key type ._llcrypt unregistered [ 889.472769] Key type ._llcrypt registered [ 889.474097] Key type .llcrypt registered [ 889.514154] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 889.969922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.989609] alg: No test for adler32 (adler32-zlib) [ 890.832991] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 890.924587] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 892.503175] Key type lgssc registered [ 892.899321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.558850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.564838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.703869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.022525] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 900.839636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.972518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 902.316048] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 904.714847] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.614061] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 908.886067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.005126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.884436] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 913.176474] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 915.426227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.428811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.432970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.018898] Lustre: server umount lustre-OST0000 complete [ 921.568018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 921.570470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.574094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.663982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.664599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.381617] Lustre: server umount lustre-MDT0000 complete [ 928.592230] LustreError: 53224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542562 with bad export cookie 7373881075580334920 [ 928.595527] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.596365] LustreError: 53224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 928.710775] Lustre: server umount lustre-MDT0001 complete [ 932.274267] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 933.282390] Key type lgssc unregistered [ 933.404388] LNet: 56776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.407534] LNetError: 56776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.416315] LNet: Removed LNI 192.168.204.131@tcp [ 933.701943] Key type .llcrypt unregistered [ 933.703189] Key type ._llcrypt unregistered [ 942.694840] Key type ._llcrypt registered [ 942.696018] Key type .llcrypt registered [ 942.731586] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 943.069717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.125084] alg: No test for adler32 (adler32-zlib) [ 943.974731] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 944.064431] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 945.647179] Key type lgssc registered [ 946.026518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.723863] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.730172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.191488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 969.311238] LustreError: 58141:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 995.935149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.844791] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1000.234943] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1002.426394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.601359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.627025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1004.015131] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1006.883328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.017845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.372344] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1010.854387] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1012.609795] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1015.268818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.271055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.272897] Lustre: Skipped 1 previous similar message [ 1019.347197] Lustre: server umount lustre-OST0000 complete [ 1033.447725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.448239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.451280] Lustre: Skipped 1 previous similar message [ 1033.453024] Lustre: Skipped 1 previous similar message [ 1035.488814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.490865] Lustre: Skipped 2 previous similar messages [ 1039.714097] Lustre: server umount lustre-MDT0000 complete [ 1040.863982] LustreError: 59181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.868604] LustreError: 59181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1040.987240] LustreError: 58148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542675 with bad export cookie 2749086666316373363 [ 1040.990616] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.991991] LustreError: 58148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1047.157313] Lustre: server umount lustre-MDT0001 complete [ 1050.815475] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1051.874933] Key type lgssc unregistered [ 1052.003307] LNet: 61883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.005718] LNetError: 61883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.015354] LNet: Removed LNI 192.168.204.131@tcp [ 1052.326403] Key type .llcrypt unregistered [ 1052.327817] Key type ._llcrypt unregistered [ 1061.388099] Key type ._llcrypt registered [ 1061.389223] Key type .llcrypt registered [ 1061.424575] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1061.823084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.829643] alg: No test for adler32 (adler32-zlib) [ 1062.675127] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1062.763912] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1064.343103] Key type lgssc registered [ 1064.696526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.287540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.295481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.462710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.929732] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1072.898971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.034722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.348872] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1076.695382] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.519316] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1080.660066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.780960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.633070] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1085.051697] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.094721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1098.213265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.213638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.215235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.221691] Lustre: Skipped 1 previous similar message [ 1099.604551] Lustre: server umount lustre-OST0000 complete [ 1102.938676] Lustre: server umount lustre-MDT0000 complete [ 1103.329408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.330274] LustreError: 63266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.333535] Lustre: Skipped 1 previous similar message [ 1103.338604] LustreError: 63266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1104.154245] LustreError: 63252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542738 with bad export cookie 8528979281571271772 [ 1104.157055] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.160140] LustreError: 63252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1104.273690] Lustre: server umount lustre-MDT0001 complete [ 1107.918171] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1108.922833] Key type lgssc unregistered [ 1109.051395] LNet: 66822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.053674] LNetError: 66822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.060327] LNet: Removed LNI 192.168.204.131@tcp [ 1109.364693] Key type .llcrypt unregistered [ 1109.365886] Key type ._llcrypt unregistered [ 1118.778413] Key type ._llcrypt registered [ 1118.779635] Key type .llcrypt registered [ 1118.821541] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1119.267595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1119.298582] alg: No test for adler32 (adler32-zlib) [ 1120.158169] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1120.256300] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1121.839264] Key type lgssc registered [ 1122.173059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1125.096364] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:52:39 (1786542759) [ 1127.308807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1127.321118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.506290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.754457] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1132.639795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.765524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1134.106933] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1136.832305] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1138.655204] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1140.839636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1140.949937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.027174] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1145.534665] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1154.279099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1158.381378] Lustre: Failing over lustre-MDT0000 [ 1158.625391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1158.627881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.632607] LustreError: 70338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.637835] LustreError: 70338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1158.650266] Lustre: server umount lustre-MDT0000 complete [ 1159.648525] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.649485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.654978] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1159.660841] Lustre: Skipped 1 previous similar message [ 1161.378708] LustreError: 68497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.393061] LustreError: 68497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1161.773872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.812315] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.867627] Lustre: *** cfs_fail_loc=135, val=0*** [ 1161.911105] LustreError: 71608:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1161.913316] LustreError: 71608:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -2 [ 1161.915932] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1161.915932] [ 1161.918632] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1161.922375] LustreError: 71596:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1161.924858] LustreError: 71596:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1161.927446] LustreError: 71596:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1161.929266] LustreError: 71596:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1161.940517] Lustre: server umount lustre-MDT0000 complete [ 1161.941981] LustreError: 71596:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1164.163325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.202963] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.305535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.337829] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1165.760728] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1166.483041] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1169.385092] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1169.396244] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1169.419002] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1172.959913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1172.962367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.966521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.499114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.503176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1176.531444] Lustre: server umount lustre-OST0000 complete [ 1179.617111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.617562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1179.617779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.617784] Lustre: Skipped 1 previous similar message [ 1179.621861] Lustre: Skipped 2 previous similar messages [ 1184.046777] Lustre: server umount lustre-MDT0000 complete [ 1184.736065] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.740809] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1185.334342] LustreError: 68483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542819 with bad export cookie 17641715880363744005 [ 1185.337394] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.339529] LustreError: 68483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1185.465082] Lustre: server umount lustre-MDT0001 complete [ 1189.067898] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1190.096601] Key type lgssc unregistered [ 1190.219448] LNet: 73384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.222259] LNetError: 73384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.228346] LNet: Removed LNI 192.168.204.131@tcp [ 1190.622615] Key type .llcrypt unregistered [ 1190.623773] Key type ._llcrypt unregistered [ 1199.032860] Key type ._llcrypt registered [ 1199.034009] Key type .llcrypt registered [ 1199.068820] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1199.375886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1199.409163] alg: No test for adler32 (adler32-zlib) [ 1200.260354] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1200.349183] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1201.927191] Key type lgssc registered [ 1202.274644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1204.692396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1204.697944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.843074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.106525] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1209.859644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.997205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.325785] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1213.577150] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.452932] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1217.660057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1217.771057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.606681] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1221.914888] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1230.054200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1234.649915] Lustre: Failing over lustre-MDT0000 [ 1234.841313] Lustre: server umount lustre-MDT0000 complete [ 1235.425168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.426831] LustreError: 74770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.428655] Lustre: Skipped 2 previous similar messages [ 1235.434273] LustreError: 74770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1237.657234] LustreError: 76612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.824877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.867244] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.923773] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1237.982596] LustreError: 77877:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1237.985751] LustreError: 77877:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -9 [ 1237.988567] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1237.988567] [ 1237.991420] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1237.998312] LustreError: 77865:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1238.002306] LustreError: 77865:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1238.007558] LustreError: 77865:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1238.011543] LustreError: 77865:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1238.026346] Lustre: server umount lustre-MDT0000 complete [ 1238.028839] LustreError: 77865:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1240.115213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.152365] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.241555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.268837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1241.582817] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1242.770810] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1245.672927] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1245.681830] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1245.700271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1250.791406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.791557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.792118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.795032] Lustre: Skipped 1 previous similar message [ 1252.813793] Lustre: server umount lustre-OST0000 complete [ 1255.905653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.906301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.907371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.907391] Lustre: Skipped 1 previous similar message [ 1255.911046] Lustre: Skipped 1 previous similar message [ 1260.191854] Lustre: server umount lustre-MDT0000 complete [ 1261.023947] LustreError: 75499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.028373] LustreError: 75499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1261.385555] LustreError: 75507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542895 with bad export cookie 6505216392638329789 [ 1261.390902] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.392357] LustreError: 75507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1261.504475] Lustre: server umount lustre-MDT0001 complete [ 1265.179145] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1266.166037] Key type lgssc unregistered [ 1266.286494] LNet: 79656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.290668] LNetError: 79656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.300442] LNet: Removed LNI 192.168.204.131@tcp [ 1266.629910] Key type .llcrypt unregistered [ 1266.631754] Key type ._llcrypt unregistered [ 1270.901926] Key type ._llcrypt registered [ 1270.902965] Key type .llcrypt registered [ 1270.943455] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1271.058323] Key type .llcrypt unregistered [ 1271.059514] Key type ._llcrypt unregistered [ 1280.035593] Key type ._llcrypt registered [ 1280.037078] Key type .llcrypt registered [ 1280.076370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1280.385597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.404082] alg: No test for adler32 (adler32-zlib) [ 1281.250611] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1281.333306] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1282.911167] Key type lgssc registered [ 1283.288503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1285.686452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1285.694881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.850743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.108277] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1290.936759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.059731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1292.277116] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1294.641686] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1296.382543] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1298.625642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1298.741358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.657508] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1302.922436] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1311.015048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1316.324354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.325060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1316.325739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.328845] Lustre: Skipped 1 previous similar message [ 1321.441549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1321.443516] Lustre: Skipped 2 previous similar messages [ 1322.257426] Lustre: server umount lustre-OST0000 complete [ 1326.560660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.561315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.564020] Lustre: Skipped 1 previous similar message [ 1326.566881] Lustre: Skipped 1 previous similar message [ 1329.565673] Lustre: server umount lustre-MDT0000 complete [ 1330.733542] LustreError: 81518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542964 with bad export cookie 10644219551631868505 [ 1330.736524] LustreError: 81518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1330.736730] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1330.858654] Lustre: server umount lustre-MDT0001 complete [ 1335.135659] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1336.248524] Key type lgssc unregistered [ 1336.370400] LNet: 85092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.372604] LNetError: 85092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.380311] LNet: Removed LNI 192.168.204.131@tcp [ 1336.695478] Key type .llcrypt unregistered [ 1336.696522] Key type ._llcrypt unregistered [ 1345.578362] Key type ._llcrypt registered [ 1345.579485] Key type .llcrypt registered [ 1345.621152] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1346.092886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1346.123579] alg: No test for adler32 (adler32-zlib) [ 1346.965052] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1347.048400] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1348.631142] Key type lgssc registered [ 1348.989691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.469535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.475693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.612964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.882657] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1356.553179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.664517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.890821] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1360.286767] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.077642] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.292910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.407921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.198942] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1368.515632] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.705832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1381.855763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.858022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.861567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.881359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.881387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.882167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.891425] Lustre: Skipped 1 previous similar message [ 1387.335359] Lustre: server umount lustre-OST0000 complete [ 1392.608143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.611436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.616820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.642243] Lustre: server umount lustre-MDT0000 complete [ 1395.850843] LustreError: 89282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543029 with bad export cookie 2178189504974279050 [ 1395.854211] LustreError: 89282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1395.856756] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.984190] Lustre: server umount lustre-MDT0001 complete [ 1399.880961] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1401.132314] Key type lgssc unregistered [ 1401.265547] LNet: 90035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.268625] LNetError: 90035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.275537] LNet: Removed LNI 192.168.204.131@tcp [ 1401.634098] Key type .llcrypt unregistered [ 1401.636274] Key type ._llcrypt unregistered [ 1410.442437] Key type ._llcrypt registered [ 1410.443517] Key type .llcrypt registered [ 1410.482548] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1410.835849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.864793] alg: No test for adler32 (adler32-zlib) [ 1411.702779] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1411.784822] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1413.359128] Key type lgssc registered [ 1413.702641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1416.312849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1416.318335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.495790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.834827] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1421.535058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.652956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1422.910756] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1425.187232] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1426.933820] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1429.091510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1429.216021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.121211] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1433.515727] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.539123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1447.905659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.906020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.912602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.914528] Lustre: Skipped 1 previous similar message [ 1453.024918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.027653] Lustre: Skipped 1 previous similar message [ 1453.129408] Lustre: server umount lustre-OST0000 complete [ 1457.631957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1457.634581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.638324] Lustre: Skipped 1 previous similar message [ 1457.640316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.760070] Lustre: server umount lustre-MDT0000 complete [ 1461.953243] LustreError: 91405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543096 with bad export cookie 11846235295689181027 [ 1461.956661] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.958816] LustreError: 91405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1462.071539] Lustre: server umount lustre-MDT0001 complete [ 1466.243497] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1467.317142] Key type lgssc unregistered [ 1467.442457] LNet: 94979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.445122] LNetError: 94979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.455314] LNet: Removed LNI 192.168.204.131@tcp [ 1467.755242] Key type .llcrypt unregistered [ 1467.756369] Key type ._llcrypt unregistered [ 1476.884339] Key type ._llcrypt registered [ 1476.885570] Key type .llcrypt registered [ 1476.920390] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1477.306278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1477.319055] alg: No test for adler32 (adler32-zlib) [ 1478.162452] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1478.244418] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1479.823167] Key type lgssc registered [ 1480.169908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1482.860992] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1482.870402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1498.335331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1503.455250] LustreError: 96342:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 1530.079177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.340629] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1534.842448] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1536.667630] Lustre: server umount lustre-OST0000 complete [ 1539.349414] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:59:33 (1786543173) [ 1541.661493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.806908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.142958] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1545.930147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.044745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1547.448719] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1550.045831] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1551.898982] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1554.185358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.304335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.143751] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1558.451738] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.637302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1576.933961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.936140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.936249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1576.939504] Lustre: Skipped 1 previous similar message [ 1578.581106] Lustre: server umount lustre-OST0000 complete [ 1582.050837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1582.051877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.054520] LustreError: Skipped 1 previous similar message [ 1582.065340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.068182] Lustre: Skipped 2 previous similar messages [ 1587.169862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.172368] Lustre: Skipped 1 previous similar message [ 1592.288724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.291421] Lustre: Skipped 1 previous similar message [ 1594.335181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1594.417687] Lustre: server umount lustre-MDT0000 complete [ 1595.879538] LustreError: 97958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543229 with bad export cookie 8303044596659305987 [ 1595.884968] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.887293] LustreError: 97958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1596.056103] Lustre: server umount lustre-MDT0001 complete [ 1599.735338] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1600.929996] Key type lgssc unregistered [ 1601.104592] LNet: 101691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.108360] LNetError: 101691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.117732] LNet: Removed LNI 192.168.204.131@tcp [ 1601.570525] Key type .llcrypt unregistered [ 1601.572634] Key type ._llcrypt unregistered [ 1618.051939] Key type ._llcrypt registered [ 1618.053258] Key type .llcrypt registered [ 1618.088477] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1618.388021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1618.417108] alg: No test for adler32 (adler32-zlib) [ 1619.276949] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1619.370789] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1620.951195] Key type lgssc registered [ 1621.499943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1625.232740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1625.243417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.405496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.114991] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1632.264316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.449531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1634.471914] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1637.966682] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1640.540219] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1643.486429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1643.619431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.651904] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1648.464092] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.819184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1662.433577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.442063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.454655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.459942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.461440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.469388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.812374] Lustre: server umount lustre-OST0000 complete [ 1673.698605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1673.698639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.700000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.700009] Lustre: Skipped 1 previous similar message [ 1673.722248] Lustre: Skipped 1 previous similar message [ 1675.821507] Lustre: server umount lustre-MDT0000 complete [ 1677.301174] LustreError: 103877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543311 with bad export cookie 10772827786088039903 [ 1677.306352] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.310476] LustreError: 103877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1677.455744] Lustre: server umount lustre-MDT0001 complete [ 1682.235173] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1683.745974] Key type lgssc unregistered [ 1683.897636] LNet: 107416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.902346] LNetError: 107416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.910508] LNet: Removed LNI 192.168.204.131@tcp [ 1684.367953] Key type .llcrypt unregistered [ 1684.370260] Key type ._llcrypt unregistered [ 1693.503592] Key type ._llcrypt registered [ 1693.504947] Key type .llcrypt registered [ 1693.548784] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1693.886230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.933757] alg: No test for adler32 (adler32-zlib) [ 1694.787958] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1694.878836] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1696.463160] Key type lgssc registered [ 1697.028439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1700.511976] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1700.524216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1715.999295] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1721.119239] LustreError: 108636:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 1747.743273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.475715] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1753.821259] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.917867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.008204] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1757.012894] LustreError: 109660:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1757.015653] LustreError: 109660:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1757.018471] LustreError: 109660:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1757.044418] Lustre: server umount lustre-MDT0000 complete [ 1757.046201] LustreError: 109660:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1770.506876] Lustre: server umount lustre-OST0000 complete [ 1774.511472] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1778.728316] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1794.845445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.978681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1794.993258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1795.034662] Lustre: lustre-MDT0000: new disk, initializing [ 1795.057828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.065747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1796.660218] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1801.908560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.978428] Lustre: 112449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1802.017883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1802.021299] Lustre: Skipped 1 previous similar message [ 1802.064371] Lustre: lustre-MDT0001: new disk, initializing [ 1802.107792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.128202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1802.133956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1803.769909] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1807.886422] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.448305] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1813.586795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1813.704625] Lustre: lustre-OST0000: new disk, initializing [ 1813.706516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1813.708968] Lustre: 114416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1813.735411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.600977] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1821.421069] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1822.223682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1822.231559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1822.276374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1823.394864] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1823.505342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1825.356600] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1825.478577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1827.301174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1827.302637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.304336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1827.318337] Lustre: Skipped 1 previous similar message [ 1828.525160] Lustre: server umount lustre-OST0000 complete [ 1832.420178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.421607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1832.427974] Lustre: Skipped 1 previous similar message [ 1832.435413] Lustre: Skipped 2 previous similar messages [ 1836.487259] Lustre: server umount lustre-MDT0000 complete [ 1838.380592] LustreError: 112442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543472 with bad export cookie 822202333965977937 [ 1838.386921] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.389271] LustreError: 112442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1838.568355] Lustre: server umount lustre-MDT0001 complete [ 1842.593416] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:04:36 (1786543476) [ 1844.252263] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1851.618947] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1855.452239] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1870.062813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.147496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1870.156607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1870.191230] Lustre: lustre-MDT0000: new disk, initializing [ 1870.210780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.216565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1871.659923] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1876.604156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.685225] Lustre: 119311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1876.731848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1876.735936] Lustre: Skipped 1 previous similar message [ 1876.782654] Lustre: lustre-MDT0001: new disk, initializing [ 1876.846532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1876.853082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1878.390620] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1882.971807] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1885.148473] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.303955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.431263] Lustre: lustre-OST0000: new disk, initializing [ 1888.433410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1888.435937] Lustre: 121274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1888.469630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.474261] Lustre: Skipped 1 previous similar message [ 1890.944709] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1894.908100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1894.912365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1894.949249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1895.249776] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1897.016901] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1897.110010] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1898.861966] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1898.960650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1900.003150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.003972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.005731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.007078] LustreError: Skipped 2 previous similar messages [ 1900.013426] Lustre: Skipped 1 previous similar message [ 1901.992785] Lustre: server umount lustre-OST0000 complete [ 1905.122134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.123535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.130576] Lustre: Skipped 1 previous similar message [ 1905.137277] Lustre: Skipped 2 previous similar messages [ 1910.000211] Lustre: server umount lustre-MDT0000 complete [ 1910.240965] LustreError: 119320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.252855] LustreError: 119320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1911.816158] LustreError: 119301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543545 with bad export cookie 822202333965978826 [ 1911.822919] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.823994] LustreError: 119301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1911.994587] Lustre: server umount lustre-MDT0001 complete [ 1916.091926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.305543] LustreError: 123487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.343098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.008481] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1921.504962] LustreError: 123488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.196324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.147383] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1927.615678] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.266252] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.451378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1936.285372] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1939.839786] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1952.806643] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1956.832828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1956.835191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.838870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.752616] Lustre: server umount lustre-OST0000 complete [ 1963.488661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1963.491009] LustreError: Skipped 1 previous similar message [ 1968.608628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.610695] Lustre: Skipped 5 previous similar messages [ 1975.263058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1975.327678] Lustre: server umount lustre-MDT0000 complete [ 1976.629613] LustreError: 123469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543610 with bad export cookie 822202333965979708 [ 1976.635099] LustreError: 123469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1976.635310] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.766440] Lustre: server umount lustre-MDT0001 complete [ 1980.962526] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1981.969279] Key type lgssc unregistered [ 1982.102393] LNet: 127140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.104611] LNetError: 127140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.112488] LNet: Removed LNI 192.168.204.131@tcp [ 1982.467699] Key type .llcrypt unregistered [ 1982.468930] Key type ._llcrypt unregistered [ 1989.267317] Key type ._llcrypt registered [ 1989.268320] Key type .llcrypt registered [ 1989.310471] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1996.005134] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1996.295917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.302864] alg: No test for adler32 (adler32-zlib) [ 1997.155835] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1997.236525] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1998.807131] Key type lgssc registered [ 1999.218302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.149300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.162840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.266024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2014.277579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2014.320339] Lustre: lustre-MDT0000: new disk, initializing [ 2014.344975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.352857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2015.883964] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2020.603148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.649638] Lustre: 130263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2020.677785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2020.680328] Lustre: Skipped 1 previous similar message [ 2020.715534] Lustre: lustre-MDT0001: new disk, initializing [ 2020.750991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.764842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2020.768263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2022.141385] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2026.002632] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2027.861710] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2030.018097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2030.162279] Lustre: lustre-OST0000: new disk, initializing [ 2030.165672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2030.170320] Lustre: 132233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2030.219747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.063882] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2036.218967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2036.225170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2036.258057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2036.273993] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.615775] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2037.693537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2039.406674] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2039.475983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2041.315056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2041.317109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.319951] LustreError: Skipped 1 previous similar message [ 2041.325762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2041.330415] Lustre: Skipped 1 previous similar message [ 2042.324718] Lustre: server umount lustre-OST0000 complete [ 2046.433933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.434683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.439501] Lustre: Skipped 2 previous similar messages [ 2046.444342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2049.832920] Lustre: server umount lustre-MDT0000 complete [ 2051.212426] LustreError: 130258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543685 with bad export cookie 13831021676895855303 [ 2051.215250] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.216954] LustreError: 130258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2051.342729] Lustre: server umount lustre-MDT0001 complete [ 2054.287338] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:08:08 (1786543688) [ 2056.450112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.628520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.064449] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2061.793429] LustreError: 134638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.800077] LustreError: 134638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2061.812897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.921046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2063.278909] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2065.857989] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.063977] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.522301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.523838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.530753] Lustre: Skipped 1 previous similar message [ 2071.538844] Lustre: Skipped 2 previous similar messages [ 2077.665109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.667461] Lustre: Skipped 3 previous similar messages [ 2082.785589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.788365] Lustre: Skipped 1 previous similar message [ 2083.295164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2083.368137] Lustre: server umount lustre-MDT0000 complete [ 2084.595065] LustreError: 135374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543718 with bad export cookie 13831021676895856171 [ 2084.597309] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.598363] LustreError: 135374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2084.704981] Lustre: server umount lustre-MDT0001 complete [ 2087.472224] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:08:41 (1786543721) [ 2089.742323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.183781] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2110.303576] LustreError: 137054:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2135.903681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.629944] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2142.168616] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2143.242128] Lustre: server umount lustre-OST0000 complete [ 2146.940827] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:09:41 (1786543781) [ 2149.888705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.096205] LustreError: 138497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.734132] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2155.488967] LustreError: 138498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.752897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.907780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2155.911788] Lustre: Skipped 1 previous similar message [ 2157.512342] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2160.820434] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2163.127602] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2166.335921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2168.876227] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2172.029155] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2186.211531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.214112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2186.217782] Lustre: Skipped 2 previous similar messages [ 2186.221727] Lustre: Skipped 1 previous similar message [ 2191.855466] Lustre: server umount lustre-MDT0000 complete [ 2193.675097] LustreError: 138477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543827 with bad export cookie 13831021676895856822 [ 2193.680385] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2193.683274] LustreError: 138477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2193.839557] Lustre: server umount lustre-MDT0001 complete [ 2205.869800] Lustre: server umount lustre-OST0000 complete [ 2209.539920] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:10:43 (1786543843) [ 2212.711200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.938826] LustreError: 142168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.977449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.980062] Lustre: Skipped 1 previous similar message [ 2214.844376] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2218.465263] LustreError: 142169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.488987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.369116] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2223.626630] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2226.007710] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2229.186302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.285537] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2234.565648] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2236.192640] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2238.509687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2239.399339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2241.100580] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2244.293683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 2248.675472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2248.676146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.679234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.679247] Lustre: Skipped 3 previous similar messages [ 2248.680519] LustreError: Skipped 1 previous similar message [ 2248.689041] Lustre: Skipped 1 previous similar message [ 2251.097961] Lustre: server umount lustre-OST0000 complete [ 2258.600781] Lustre: server umount lustre-MDT0000 complete [ 2258.912845] LustreError: 145251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.923377] LustreError: 145251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2260.422312] LustreError: 145525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543894 with bad export cookie 13831021676895857872 [ 2260.429060] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.430208] LustreError: 145525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2260.614529] Lustre: server umount lustre-MDT0001 complete [ 2264.450530] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:11:38 (1786543898) [ 2267.563974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2283.039738] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2288.159525] LustreError: 146370:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2313.759320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.761829] Lustre: Skipped 2 previous similar messages [ 2316.125591] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2319.434257] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2322.571604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.778828] LustreError: 146395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.849622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2324.548288] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2328.401041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.301834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2333.852246] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2336.496527] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2338.334607] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2338.405395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2340.097702] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2340.176083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2343.905717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.909993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.916480] Lustre: Skipped 2 previous similar messages [ 2343.919544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.922518] Lustre: Skipped 3 previous similar messages [ 2347.171884] Lustre: server umount lustre-MDT0000 complete [ 2348.636191] LustreError: 146376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543982 with bad export cookie 13831021676895858887 [ 2348.641383] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2348.786831] Lustre: server umount lustre-MDT0001 complete [ 2360.376558] Lustre: server umount lustre-OST0000 complete [ 2364.364068] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:13:18 (1786543998) [ 2367.736475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.199833] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2388.319311] LustreError: 150568:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2416.410182] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2419.663321] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.502962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.741119] LustreError: 150593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.752249] LustreError: 150593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2422.812081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2424.577907] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2428.582585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.530735] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2433.755267] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2436.254834] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2439.222565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.310123] Lustre: lustre-OST0001: new disk, initializing [ 2439.313377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2439.316790] Lustre: 153438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2439.877742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2439.883674] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2439.925671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2442.056101] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2446.870735] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2448.626277] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2448.708384] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2450.361085] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2450.448890] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2453.471911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2453.475895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.480845] Lustre: Skipped 2 previous similar messages [ 2453.483397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.485914] Lustre: Skipped 3 previous similar messages [ 2457.286235] Lustre: server umount lustre-OST0000 complete [ 2465.119295] Lustre: server umount lustre-OST0001 complete [ 2469.857686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2469.863270] LustreError: Skipped 3 previous similar messages [ 2472.794148] Lustre: server umount lustre-MDT0000 complete [ 2474.019248] LustreError: 150576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544108 with bad export cookie 13831021676895859839 [ 2474.021830] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2474.023624] LustreError: 150576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2474.141728] Lustre: server umount lustre-MDT0001 complete [ 2482.014602] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:15:16 (1786544116) [ 2482.595640] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2483.209942] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2483.817753] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:15:18 (1786544118) [ 2486.077850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.181936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2486.195404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2486.261708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.263826] Lustre: Skipped 6 previous similar messages [ 2487.666230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2490.519879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.555888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2490.565244] Lustre: 156992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2492.099416] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2494.659705] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2496.578483] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2498.781445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.867248] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2500.825922] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2502.668476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2503.230953] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2504.615849] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2504.695711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2506.056418] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2506.126213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2507.745142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.747527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.748967] LustreError: Skipped 1 previous similar message [ 2507.756915] Lustre: Skipped 6 previous similar messages [ 2508.876767] Lustre: server umount lustre-OST0000 complete [ 2522.395156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.310247] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2526.708179] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2527.652763] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.657059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.657127] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2527.665121] Lustre: Skipped 1 previous similar message [ 2528.167724] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2528.229083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2529.405795] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2529.470192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2531.236766] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94bd07f55000.ost_server_uuid 50 [ 2531.756996] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff94bd07f55000.ost_server_uuid in FULL state after 0 sec [ 2531.826269] LustreError: 157027:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.31@tcp) returned error from blocking AST (req@00000000ce42720a x1873326774608384 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000007d902228/0xbff1a8ef6c1afdcb lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.31@tcp remote: 0xf8a446661fc97340 expref: 5 pid: 157027 timeout: 2631 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2531.839038] LustreError: lustre-MDT0000: A client on nid 192.168.204.31@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2531.842366] LustreError: 156987:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.31@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000007d902228/0xbff1a8ef6c1afdcb lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.31@tcp remote: 0xf8a446661fc97340 expref: 6 pid: 157027 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2547.167165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2547.221187] Lustre: server umount lustre-OST0000 complete [ 2552.287800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2552.291604] LustreError: Skipped 2 previous similar messages [ 2554.649983] Lustre: server umount lustre-MDT0000 complete [ 2555.862929] LustreError: 157028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544189 with bad export cookie 13831021676895861022 [ 2555.864911] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.866621] LustreError: 157028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2570.207126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2570.298070] Lustre: server umount lustre-MDT0001 complete [ 2574.298160] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2575.352441] Key type lgssc unregistered [ 2575.488408] LNet: 163098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2575.492131] LNetError: 163098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2575.500420] LNet: Removed LNI 192.168.204.131@tcp [ 2575.841156] Key type .llcrypt unregistered [ 2575.842297] Key type ._llcrypt unregistered [ 2584.983665] Key type ._llcrypt registered [ 2584.984795] Key type .llcrypt registered [ 2585.023813] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2585.479129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2585.498208] alg: No test for adler32 (adler32-zlib) [ 2586.360190] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2586.456077] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2588.047145] Key type lgssc registered [ 2588.414662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2591.037166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2591.043270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.190830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.538045] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2596.348854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.466330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.845623] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2600.268913] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2602.124797] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2604.487170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2604.616151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.483029] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2608.920167] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2617.954745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2621.862586] Lustre: Failing over lustre-MDT0000 [ 2622.059402] Lustre: server umount lustre-MDT0000 complete [ 2622.432562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2622.436409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.445366] LustreError: 164487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.455241] LustreError: 164487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2623.458118] LustreError: 164486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.460777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.468225] LustreError: 164486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2623.476042] Lustre: Skipped 1 previous similar message [ 2628.576955] LustreError: 165926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.582711] LustreError: 165926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2633.113282] LustreError: 164468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.31@tcp arrived at 1786544267 with bad export cookie 14267173281762943006 [ 2633.120487] LustreError: 164468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2633.696702] LustreError: 164487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.703844] LustreError: 164487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2636.104504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.162492] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.293260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.321684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2637.815567] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2638.546012] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2638.550611] Lustre: lustre-MDT0000: Denying connection for new client 817928dd-a555-4d70-8d50-b0a4ab134648 (at 192.168.204.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2641.385821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2646.500407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.509402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.512563] Lustre: Skipped 1 previous similar message [ 2651.598585] Lustre: server umount lustre-OST0000 complete [ 2652.753283] LustreError: 168233:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2652.757780] Lustre: 167582:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2652.760356] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2652.766385] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2652.770240] LustreError: 167582:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007f4999b8 x1873327392067200/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2652.775873] LustreError: 167582:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2652.778792] LustreError: 167582:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2652.781690] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2652.887956] Lustre: server umount lustre-MDT0000 complete [ 2654.012527] LustreError: 166328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544288 with bad export cookie 14267173281762943363 [ 2654.015692] LustreError: 166328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2654.016102] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2654.016631] LustreError: 163798:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2660.200866] Lustre: server umount lustre-MDT0001 complete [ 2663.570919] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2664.617536] Key type lgssc unregistered [ 2664.755434] LNet: 168988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.757578] LNetError: 168988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2664.766422] LNet: Removed LNI 192.168.204.131@tcp [ 2665.107210] Key type .llcrypt unregistered [ 2665.108246] Key type ._llcrypt unregistered [ 2674.025806] Key type ._llcrypt registered [ 2674.026921] Key type .llcrypt registered [ 2674.061084] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2674.416558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.450055] alg: No test for adler32 (adler32-zlib) [ 2675.291259] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2675.372743] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2676.951134] Key type lgssc registered [ 2677.268947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2679.611935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2679.617508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.753173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.980328] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2684.952465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.103513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.565151] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2689.694988] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2691.866749] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2694.038933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.151146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.973993] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2698.253706] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.444447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2710.503255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.504133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2710.507913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2710.509632] LustreError: Skipped 1 previous similar message [ 2712.081792] Lustre: server umount lustre-OST0000 complete [ 2715.420300] Lustre: server umount lustre-MDT0000 complete [ 2715.618136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.622091] Lustre: Skipped 1 previous similar message [ 2715.625130] LustreError: 171666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.634832] LustreError: 171666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2716.868685] LustreError: 170358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544350 with bad export cookie 7839953984316161473 [ 2716.872519] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.016715] Lustre: server umount lustre-MDT0001 complete [ 2721.114081] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2722.371274] Key type lgssc unregistered [ 2722.499576] LNet: 173929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.504737] LNetError: 173929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.516603] LNet: Removed LNI 192.168.204.131@tcp [ 2722.912961] Key type .llcrypt unregistered [ 2722.913991] Key type ._llcrypt unregistered [ 2735.314060] Key type ._llcrypt registered [ 2735.315136] Key type .llcrypt registered [ 2735.356747] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2735.751618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2735.802870] alg: No test for adler32 (adler32-zlib) [ 2736.654421] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2736.739375] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2738.319121] Key type lgssc registered [ 2738.654766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.102488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.110919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.258996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.441759] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2745.940945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.067046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2747.264708] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2749.594917] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2751.511154] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2753.580715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.687768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.425627] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2757.723227] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2767.014300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2772.449040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2772.452342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.455987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.457753] Lustre: Skipped 1 previous similar message [ 2775.383658] Lustre: server umount lustre-OST0000 complete [ 2777.056082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2777.059514] LustreError: Skipped 1 previous similar message [ 2777.061874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.068370] Lustre: Skipped 1 previous similar message [ 2777.071200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.689318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.693113] Lustre: Skipped 2 previous similar messages [ 2787.809413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2787.811892] Lustre: Skipped 1 previous similar message [ 2791.391121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2791.469560] Lustre: server umount lustre-MDT0000 complete [ 2792.929343] LustreError: 175610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.940139] LustreError: 175610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2793.191680] LustreError: 175590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544427 with bad export cookie 9055116661005090190 [ 2793.199899] LustreError: 175590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2793.200471] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.374163] Lustre: server umount lustre-MDT0001 complete [ 2797.326631] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2798.355083] Key type lgssc unregistered [ 2798.475545] LNet: 179163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.478024] LNetError: 179163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.487458] LNet: Removed LNI 192.168.204.131@tcp [ 2798.811494] Key type .llcrypt unregistered [ 2798.812803] Key type ._llcrypt unregistered [ 2807.593153] Key type ._llcrypt registered [ 2807.594207] Key type .llcrypt registered [ 2807.635139] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2808.012696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2808.029898] alg: No test for adler32 (adler32-zlib) [ 2808.912957] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2809.021360] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2810.607206] Key type lgssc registered [ 2811.169855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2816.033229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2816.045075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.230723] Lustre: *** cfs_fail_loc=135, val=0*** [ 2817.273980] LustreError: 180661:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2817.276363] LustreError: 180661:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -2 [ 2817.279634] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2817.279634] [ 2817.282518] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2817.286409] LustreError: 180622:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2817.288591] LustreError: 180622:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2817.290630] LustreError: 180622:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2817.292472] LustreError: 180622:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2817.324536] Lustre: server umount lustre-MDT0000 complete [ 2817.326011] LustreError: 180622:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2819.722886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.871355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.181214] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2824.317942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.453071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.807765] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2828.524632] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2830.896246] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2834.018412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.023379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.025440] Lustre: Skipped 1 previous similar message [ 2838.111642] Lustre: server umount lustre-MDT0000 complete [ 2839.367184] LustreError: 181014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544473 with bad export cookie 2882147176529643161 [ 2839.369964] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.370745] LustreError: 181014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2839.487263] Lustre: server umount lustre-MDT0001 complete [ 2843.029725] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2844.144133] Key type lgssc unregistered [ 2844.284580] LNet: 183355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2844.287214] LNetError: 183355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2844.295406] LNet: Removed LNI 192.168.204.131@tcp [ 2844.625593] Key type .llcrypt unregistered [ 2844.626881] Key type ._llcrypt unregistered [ 2853.111999] Key type ._llcrypt registered [ 2853.113896] Key type .llcrypt registered [ 2853.158200] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2853.293372] Key type .llcrypt unregistered [ 2853.295395] Key type ._llcrypt unregistered [ 2860.484634] Key type ._llcrypt registered [ 2860.485924] Key type .llcrypt registered [ 2860.538355] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2860.981657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2861.010221] alg: No test for adler32 (adler32-zlib) [ 2861.865235] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2861.954612] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2863.535144] Key type lgssc registered [ 2864.003982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2866.720264] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2866.726344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2882.143428] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2887.263448] LustreError: 185473:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2913.887309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.590007] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2919.898239] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.949479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.208879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.244765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2924.939125] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2928.356398] LustreError: 186529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.364563] LustreError: 186529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2929.006716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.199802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.144660] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2934.438055] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2936.931188] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2938.125398] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2949.088130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.091219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.097099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2949.603469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.608527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.618206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.728500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.732772] Lustre: Skipped 1 previous similar message [ 2959.844467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2959.849266] Lustre: Skipped 1 previous similar message [ 2962.399155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.479681] Lustre: server umount lustre-OST0000 complete [ 2964.963170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.963973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.965305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.984928] Lustre: Skipped 1 previous similar message [ 2970.492348] Lustre: server umount lustre-MDT0000 complete [ 2972.352513] LustreError: 188133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544606 with bad export cookie 10322929020652767189 [ 2972.363550] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.515746] Lustre: server umount lustre-MDT0001 complete [ 2977.288034] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2978.651890] Key type lgssc unregistered [ 2978.811552] LNet: 189536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.817083] LNetError: 189536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2978.828443] LNet: Removed LNI 192.168.204.131@tcp [ 2979.317437] Key type .llcrypt unregistered [ 2979.319137] Key type ._llcrypt unregistered [ 2989.798784] Key type ._llcrypt registered [ 2989.800079] Key type .llcrypt registered [ 2989.846658] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2990.290085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2990.302502] alg: No test for adler32 (adler32-zlib) [ 2991.177192] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2991.287798] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2992.871195] Key type lgssc registered [ 2993.494014] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2996.718175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2996.733810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.946088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.515761] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3003.234630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.407654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.983827] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3007.628854] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3009.545057] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3011.630091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.743711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.415158] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3015.837996] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3021.993842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3027.910212] Lustre: Failing over lustre-MDT0000 [ 3028.030159] Lustre: server umount lustre-MDT0000 complete [ 3028.960114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3028.964147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.971096] LustreError: 190926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.975135] LustreError: 190926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3031.705587] LustreError: 190923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.548895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.556734] Lustre: Skipped 1 previous similar message [ 3036.818878] LustreError: 190923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.829292] LustreError: 190923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3041.509281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.561281] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.703915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.733495] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3041.939778] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3043.267437] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3045.090295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3046.887942] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3046.897866] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3046.920089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3050.953257] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3055.071993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.074944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.079516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3057.121431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3057.121525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3057.124625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.125870] LustreError: Skipped 1 previous similar message [ 3060.051737] Lustre: server umount lustre-OST0000 complete [ 3062.243412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.243494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.244567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.244577] Lustre: Skipped 1 previous similar message [ 3062.258840] Lustre: Skipped 1 previous similar message [ 3063.472464] Lustre: server umount lustre-MDT0000 complete [ 3065.200283] LustreError: 194815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544699 with bad export cookie 12950050098551575405 [ 3065.207477] LustreError: 194815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3065.207575] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.442516] Lustre: server umount lustre-MDT0001 complete [ 3072.272516] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3073.665139] Key type lgssc unregistered [ 3073.802486] LNet: 195873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3073.807642] LNetError: 195873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3073.818476] LNet: Removed LNI 192.168.204.131@tcp [ 3074.298151] Key type .llcrypt unregistered [ 3074.300946] Key type ._llcrypt unregistered [ 3086.201467] Key type ._llcrypt registered [ 3086.203850] Key type .llcrypt registered [ 3086.261952] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3086.681964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3086.740110] alg: No test for adler32 (adler32-zlib) [ 3087.651717] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3087.786602] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3089.399229] Key type lgssc registered [ 3089.915701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3093.331510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3093.342674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.575738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.458388] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3100.785140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.986156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3102.591278] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3105.611673] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3108.358657] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3111.802682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.967862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.681628] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3118.227312] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3119.154584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3125.486413] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3133.367589] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3145.021253] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3154.912551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.917307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.925850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.937796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.938436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.947361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.712205] Lustre: server umount lustre-OST0000 complete [ 3162.592631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3162.597724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.606424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.610474] Lustre: Skipped 1 previous similar message [ 3166.177133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.178037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.547745] Lustre: server umount lustre-MDT0000 complete [ 3169.299994] LustreError: 199794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544803 with bad export cookie 2671031583650332405 [ 3169.304544] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3169.309593] LustreError: 199794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3169.457296] Lustre: server umount lustre-MDT0001 complete [ 3174.098925] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3175.592331] Key type lgssc unregistered [ 3175.747659] LNet: 201111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3175.752976] LNetError: 201111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3175.767545] LNet: Removed LNI 192.168.204.131@tcp [ 3176.232418] Key type .llcrypt unregistered [ 3176.234990] Key type ._llcrypt unregistered [ 3185.715946] Key type ._llcrypt registered [ 3185.717425] Key type .llcrypt registered [ 3185.760432] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3186.152995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3186.159233] alg: No test for adler32 (adler32-zlib) [ 3187.009603] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3187.099758] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3188.679203] Key type lgssc registered [ 3189.269968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3192.974172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3192.983382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.183610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.869574] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3199.486818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.657759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.408500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3203.935337] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3206.414227] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3208.928642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.037976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.644213] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3212.683662] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.309092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3225.214109] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3233.787113] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3242.287916] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3247.661691] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3260.388697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.389382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.391355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3260.405496] Lustre: Skipped 1 previous similar message [ 3265.505523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3265.509507] Lustre: Skipped 1 previous similar message [ 3270.625627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.183151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3273.274791] Lustre: server umount lustre-OST0000 complete [ 3275.745961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.747174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.749360] Lustre: Skipped 1 previous similar message [ 3275.753282] Lustre: Skipped 1 previous similar message [ 3280.866072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.870313] Lustre: Skipped 1 previous similar message [ 3281.207354] Lustre: server umount lustre-MDT0000 complete [ 3282.765681] LustreError: 206338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544916 with bad export cookie 10432765257494272430 [ 3282.770497] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.906661] Lustre: server umount lustre-MDT0001 complete [ 3287.384283] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3288.868397] Key type lgssc unregistered [ 3289.035586] LNet: 207783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3289.040501] LNetError: 207783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3289.049619] LNet: Removed LNI 192.168.204.131@tcp [ 3289.522661] Key type .llcrypt unregistered [ 3289.524957] Key type ._llcrypt unregistered [ 3300.080326] Key type ._llcrypt registered [ 3300.082016] Key type .llcrypt registered [ 3300.138419] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3300.487426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.522360] alg: No test for adler32 (adler32-zlib) [ 3301.442613] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3301.567172] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3303.167230] Key type lgssc registered [ 3303.741496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.335663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.347312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.552706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.351238] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3314.154931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.347919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3316.053158] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3319.409224] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3321.740418] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3324.168544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3324.332780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.527576] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3328.993803] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3333.557590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3338.721227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3338.724055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.726678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.727534] Lustre: Skipped 1 previous similar message [ 3340.018952] Lustre: server umount lustre-OST0000 complete [ 3343.841928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.846171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.847790] Lustre: Skipped 2 previous similar messages [ 3344.124993] Lustre: server umount lustre-MDT0000 complete [ 3346.042467] LustreError: 209152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544980 with bad export cookie 1070572204783216930 [ 3346.049590] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3346.049989] LustreError: 209152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3346.218188] Lustre: server umount lustre-MDT0001 complete [ 3351.483758] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3353.079394] Key type lgssc unregistered [ 3353.275833] LNet: 212852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3353.283674] LNetError: 212852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3353.295740] LNet: Removed LNI 192.168.204.131@tcp [ 3353.779937] Key type .llcrypt unregistered [ 3353.782240] Key type ._llcrypt unregistered [ 3363.215443] Key type ._llcrypt registered [ 3363.216866] Key type .llcrypt registered [ 3363.281899] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3363.692281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3363.701362] alg: No test for adler32 (adler32-zlib) [ 3364.618305] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3364.753201] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3366.359128] Key type lgssc registered [ 3366.772080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3369.707935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3369.719572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.905148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.914331] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3376.861094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.039176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.024242] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3382.718592] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3385.568541] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3389.004066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.153585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.039331] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3395.313546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3395.824522] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3402.596347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.600857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3402.608917] Lustre: Skipped 1 previous similar message [ 3407.731389] Lustre: server umount lustre-OST0000 complete [ 3412.961883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3412.962161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.963547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3412.974302] Lustre: Skipped 1 previous similar message [ 3415.809972] Lustre: server umount lustre-MDT0000 complete [ 3417.741108] LustreError: 214077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545051 with bad export cookie 17706728545252047243 [ 3417.745401] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.751224] LustreError: 214077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3417.925529] Lustre: server umount lustre-MDT0001 complete [ 3422.696230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3424.266880] Key type lgssc unregistered [ 3424.431769] LNet: 217780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3424.437414] LNetError: 217780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3424.447638] LNet: Removed LNI 192.168.204.131@tcp [ 3424.926192] Key type .llcrypt unregistered [ 3424.928617] Key type ._llcrypt unregistered [ 3434.079821] Key type ._llcrypt registered [ 3434.081576] Key type .llcrypt registered [ 3434.146500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3434.513476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3434.592699] alg: No test for adler32 (adler32-zlib) [ 3435.451143] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3435.544388] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3437.127182] Key type lgssc registered [ 3437.796190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3441.856229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3441.868725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.056196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.807214] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3448.985581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.162183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3450.922498] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3454.507730] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3457.286772] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3460.549828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.726253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.352653] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3466.856753] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3468.901980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3476.791039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.820151] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3476.900188] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3478.966862] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3482.564654] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.468924] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3495.872512] Lustre: Permanently deactivating lustre-OST0001 [ 3495.882532] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3505.163954] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3510.690875] Lustre: Permanently reactivating lustre-OST0001 [ 3510.701459] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3510.707768] Lustre: Skipped 2 previous similar messages [ 3514.634792] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.643740] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3514.648373] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3514.658950] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3514.676617] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3522.023258] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.026403] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3522.032120] Lustre: Skipped 2 previous similar messages [ 3526.471987] Lustre: server umount lustre-OST0001 complete [ 3527.139602] LustreError: 221515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.151179] LustreError: 221515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3530.721176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3530.721259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.726738] LustreError: Skipped 1 previous similar message [ 3530.727658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.734772] Lustre: Skipped 1 previous similar message [ 3530.745157] Lustre: Skipped 2 previous similar messages [ 3532.261833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3532.263415] LustreError: 220842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.267139] Lustre: Skipped 1 previous similar message [ 3532.277748] LustreError: 220842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3534.507375] Lustre: server umount lustre-OST0000 complete [ 3536.352193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3536.357566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.366774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3542.431789] Lustre: server umount lustre-MDT0000 complete [ 3542.496551] LustreError: 219023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3542.501161] LustreError: 219023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3543.673488] LustreError: 219002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545177 with bad export cookie 17021969499557580339 [ 3543.677823] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3543.789147] Lustre: server umount lustre-MDT0001 complete [ 3547.490520] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3548.609293] Key type lgssc unregistered [ 3548.748403] LNet: 224096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3548.752617] LNetError: 224096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3548.767429] LNet: Removed LNI 192.168.204.131@tcp [ 3549.127078] Key type .llcrypt unregistered [ 3549.128229] Key type ._llcrypt unregistered [ 3563.221856] Key type ._llcrypt registered [ 3563.222940] Key type .llcrypt registered [ 3563.260058] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3563.807256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.816180] alg: No test for adler32 (adler32-zlib) [ 3564.681860] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3564.790501] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3566.383163] Key type lgssc registered [ 3566.779346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3569.411706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3569.418238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.512284] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3570.521592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3570.582544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.155389] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3575.309743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.342669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3575.352399] Lustre: 226275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3575.463759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.996757] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3579.279245] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3581.014594] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3583.367615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.444896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3583.538689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.681100] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3588.599717] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3589.174907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3595.395477] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3597.984143] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3605.758493] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3611.314718] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3629.777881] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3629.782568] Lustre: Skipped 1 previous similar message [ 3648.386829] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3648.389105] Lustre: Skipped 2 previous similar messages [ 3686.405229] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3686.409778] Lustre: Skipped 4 previous similar messages [ 3757.330206] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3757.337429] Lustre: Skipped 7 previous similar messages [ 3763.176479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.178787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.182102] LustreError: Skipped 1 previous similar message [ 3763.193068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.200687] Lustre: Skipped 1 previous similar message [ 3765.490613] Lustre: server umount lustre-OST0000 complete [ 3768.290389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.292301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3768.300194] Lustre: Skipped 2 previous similar messages [ 3773.409630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.413869] Lustre: Skipped 2 previous similar messages [ 3773.494541] Lustre: server umount lustre-MDT0000 complete [ 3775.342103] LustreError: 227016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545409 with bad export cookie 12427705578459093388 [ 3775.348256] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3775.350284] LustreError: 227016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3775.512977] Lustre: server umount lustre-MDT0001 complete [ 3780.584581] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3782.010435] Key type lgssc unregistered [ 3782.181738] LNet: 231114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3782.187093] LNetError: 231114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3782.197663] LNet: Removed LNI 192.168.204.131@tcp [ 3782.716341] Key type .llcrypt unregistered [ 3782.719685] Key type ._llcrypt unregistered [ 3794.921634] Key type ._llcrypt registered [ 3794.923644] Key type .llcrypt registered [ 3795.000054] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3795.519633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3795.563386] alg: No test for adler32 (adler32-zlib) [ 3796.422761] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3796.525334] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3798.119188] Key type lgssc registered [ 3798.706919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3802.391310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3802.398513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.616497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.195650] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3809.186368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.351967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3811.067477] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3814.846754] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3817.665248] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3820.749793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3820.908949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.726113] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3826.961800] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3828.076179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3838.431961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3838.435624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.440761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3839.462636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3839.462769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3839.467938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.686024] Lustre: server umount lustre-OST0000 complete [ 3849.698075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.698930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3849.706270] Lustre: Skipped 1 previous similar message [ 3849.710410] Lustre: Skipped 1 previous similar message [ 3851.376345] Lustre: server umount lustre-MDT0000 complete [ 3853.161333] LustreError: 232484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545487 with bad export cookie 15056177979753338388 [ 3853.166235] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.169195] LustreError: 232484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3853.327817] Lustre: server umount lustre-MDT0001 complete [ 3857.716634] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3859.195654] Key type lgssc unregistered [ 3859.359744] LNet: 236188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3859.362566] LNetError: 236188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3859.376650] LNet: Removed LNI 192.168.204.131@tcp [ 3859.830701] Key type .llcrypt unregistered [ 3859.833129] Key type ._llcrypt unregistered [ 3870.600059] Key type ._llcrypt registered [ 3870.601250] Key type .llcrypt registered [ 3870.634031] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3871.079030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3871.092993] alg: No test for adler32 (adler32-zlib) [ 3871.944850] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3872.048751] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3873.631201] Key type lgssc registered [ 3874.232654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4141.474934] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4142.831850] Key type lgssc unregistered [ 4142.988425] LNet: 238191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4142.993569] LNetError: 238191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4143.005485] LNet: Removed LNI 192.168.204.131@tcp [ 4143.463553] Key type .llcrypt unregistered [ 4143.465806] Key type ._llcrypt unregistered [ 4154.473955] Key type ._llcrypt registered [ 4154.475553] Key type .llcrypt registered [ 4154.527228] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4161.624251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4162.031566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4162.075081] alg: No test for adler32 (adler32-zlib) [ 4162.965810] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4163.081301] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4164.687168] Key type lgssc registered [ 4165.288529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4181.798817] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4061 sec ========== 10:43:35 (1786545815) [ 4182.748896] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:43:36 (1786545816) === [ 4184.532471] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:43:38 (1786545818) === [ 4203.890414] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4205.337311] Key type lgssc unregistered [ 4205.467590] LNet: 243399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4205.474234] LNetError: 243399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4205.485540] LNet: Removed LNI 192.168.204.131@tcp [ 4205.904734] Key type .llcrypt unregistered [ 4205.907050] Key type ._llcrypt unregistered