[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1292166478 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002643] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.010000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011018] pid_max: default: 32768 minimum: 301 [ 0.012184] LSM: Security Framework initializing [ 0.015026] Yama: becoming mindful. [ 0.016153] SELinux: Initializing. [ 0.017095] *** VALIDATE selinux *** [ 0.035129] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.042497] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.043205] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.044159] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.046006] *** VALIDATE tmpfs *** [ 0.048278] *** VALIDATE proc *** [ 0.049826] *** VALIDATE cgroup *** [ 0.050014] *** VALIDATE cgroup2 *** [ 0.051395] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.052241] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.053011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.055063] Spectre V2 : User space: Vulnerable [ 0.056012] Speculative Store Bypass: Vulnerable [ 0.061014] debug: unmapping init [mem 0xffffffffbba59000-0xffffffffbba60fff] [ 0.064366] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.067382] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.068024] ... version: 2 [ 0.069012] ... bit width: 48 [ 0.070013] ... generic registers: 4 [ 0.071028] ... value mask: 0000ffffffffffff [ 0.072013] ... max period: 00007fffffffffff [ 0.073015] ... fixed-purpose events: 3 [ 0.074017] ... event mask: 000000070000000f [ 0.075350] rcu: Hierarchical SRCU implementation. [ 0.078827] smp: Bringing up secondary CPUs ... [ 0.080040] x86: Booting SMP configuration: [ 0.081014] .... node #0, CPUs: #1 #2 #3 [ 0.090372] smp: Brought up 1 node, 4 CPUs [ 0.092028] smpboot: Max logical packages: 1 [ 0.093016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.160948] node 0 deferred pages initialised in 65ms [ 0.169196] devtmpfs: initialized [ 0.172572] x86/mm: Memory block size: 128MB [ 0.181374] gcov: version magic: 0x41383552 [ 0.187089] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.196111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.202876] pinctrl core: initialized pinctrl subsystem [ 0.207608] [ 0.209035] ************************************************************* [ 0.214023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220021] ** ** [ 0.226021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233018] ** ** [ 0.237019] ** This means that this kernel is built to expose internal ** [ 0.242030] ** IOMMU data structures, which may compromise security on ** [ 0.248020] ** your system. ** [ 0.252018] ** ** [ 0.266023] ** If you see this message and you are not debugging the ** [ 0.272030] ** kernel, report this immediately to your vendor! ** [ 0.278105] ** ** [ 0.286030] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.295019] ************************************************************* [ 0.303168] NET: Registered protocol family 16 [ 0.310085] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.320190] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.327065] cpuidle: using governor menu [ 0.329000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.335029] PCI: Using configuration type 1 for base access [ 0.342172] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.361130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.364111] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.372067] cryptd: max_cpu_qlen set to 1000 [ 0.376426] ACPI: Added _OSI(Module Device) [ 0.379056] ACPI: Added _OSI(Processor Device) [ 0.383019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.386015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.395785] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.411136] ACPI: Interpreter enabled [ 0.418196] ACPI: PM: (supports S0 S3 S4 S5) [ 0.422019] ACPI: Using IOAPIC for interrupt routing [ 0.430132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.439873] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.464053] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.466047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.473126] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.483093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.496297] acpiphp: Slot [2] registered [ 0.501038] acpiphp: Slot [5] registered [ 0.503602] acpiphp: Slot [6] registered [ 0.507287] acpiphp: Slot [7] registered [ 0.523115] acpiphp: Slot [8] registered [ 0.527985] acpiphp: Slot [9] registered [ 0.531768] acpiphp: Slot [10] registered [ 0.537000] acpiphp: Slot [3] registered [ 0.541170] acpiphp: Slot [4] registered [ 0.545290] acpiphp: Slot [11] registered [ 0.550645] acpiphp: Slot [12] registered [ 0.556139] acpiphp: Slot [13] registered [ 0.560548] acpiphp: Slot [14] registered [ 0.565185] acpiphp: Slot [15] registered [ 0.568740] acpiphp: Slot [16] registered [ 0.576178] acpiphp: Slot [17] registered [ 0.580648] acpiphp: Slot [18] registered [ 0.584604] acpiphp: Slot [19] registered [ 0.590191] acpiphp: Slot [20] registered [ 0.594834] acpiphp: Slot [21] registered [ 0.599169] acpiphp: Slot [22] registered [ 0.603937] acpiphp: Slot [23] registered [ 0.608143] acpiphp: Slot [24] registered [ 0.613795] acpiphp: Slot [25] registered [ 0.617756] acpiphp: Slot [26] registered [ 0.623237] acpiphp: Slot [27] registered [ 0.628230] acpiphp: Slot [28] registered [ 0.632155] acpiphp: Slot [29] registered [ 0.636436] acpiphp: Slot [30] registered [ 0.640125] acpiphp: Slot [31] registered [ 0.644071] PCI host bridge to bus 0000:00 [ 0.648024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.655029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.663451] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.673528] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.681033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.691032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.696928] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.706095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.716854] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.739501] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.749650] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.757025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.758000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.762023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.769195] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.773842] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.775037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.779753] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.788017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.814018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.833115] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.853054] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.872020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.908255] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.973028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.003435] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.021028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.052033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.097028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.124941] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.133000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.135043] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.156000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.197968] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.226059] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.271266] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.325032] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.468830] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.613024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.621021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.648024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.674801] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.687023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.696029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.724046] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.751219] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.760527] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.761000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.761000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.761000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.761498] iommu: Default domain type: Passthrough [ 1.762000] SCSI subsystem initialized [ 1.772204] ACPI: bus type USB registered [ 1.947558] usbcore: registered new interface driver usbfs [ 1.948000] usbcore: registered new interface driver hub [ 1.950519] usbcore: registered new device driver usb [ 1.953281] pps_core: LinuxPPS API ver. 1 registered [ 1.956018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.961097] PTP clock support registered [ 1.965512] EDAC MC: Ver: 3.0.0 [ 1.970260] PCI: Using ACPI for IRQ routing [ 1.974931] NetLabel: Initializing [ 1.978013] NetLabel: domain hash size = 128 [ 1.981013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.985124] NetLabel: unlabeled traffic allowed by default [ 1.989995] vgaarb: loaded [ 1.993335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.997013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 2.013936] clocksource: Switched to clocksource kvm-clock [ 2.267801] VFS: Disk quotas dquot_6.6.0 [ 2.272158] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.277469] *** VALIDATE ramfs *** [ 2.280115] *** VALIDATE hugetlbfs *** [ 2.286315] pnp: PnP ACPI init [ 2.437895] pnp: PnP ACPI: found 6 devices [ 2.491601] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.499215] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.512168] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.522086] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.530470] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.535615] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.543289] NET: Registered protocol family 2 [ 2.550278] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.561408] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.569813] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.585890] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.593611] TCP: Hash tables configured (established 65536 bind 65536) [ 2.599506] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.605708] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.615847] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.620489] NET: Registered protocol family 1 [ 2.624610] RPC: Registered named UNIX socket transport module. [ 2.660988] RPC: Registered udp transport module. [ 2.667111] RPC: Registered tcp transport module. [ 2.672837] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.679837] NET: Registered protocol family 44 [ 2.683838] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.689168] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.695816] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.700402] PCI: CLS 0 bytes, default 64 [ 2.704300] Unpacking initramfs... [ 8.156246] debug: unmapping init [mem 0xffff9259fcc54000-0xffff9259fffbffff] [ 8.173607] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 8.177684] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 8.185387] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 10.218256] Initialise system trusted keyrings [ 10.222697] Key type blacklist registered [ 10.227679] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 10.252155] zbud: loaded [ 10.260124] *** VALIDATE nfs *** [ 10.262807] *** VALIDATE nfs4 *** [ 10.268808] pstore: using deflate compression [ 10.277485] Platform Keyring initialized [ 10.758804] NET: Registered protocol family 38 [ 10.761512] Key type asymmetric registered [ 10.765200] Asymmetric key parser 'x509' registered [ 10.770544] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 10.777380] io scheduler mq-deadline registered [ 10.782170] io scheduler kyber registered [ 10.785876] io scheduler bfq registered [ 10.790447] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 10.800665] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 10.830259] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 10.839993] ACPI: Power Button [PWRF] [ 10.934183] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 10.950222] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 10.975155] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 10.996993] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 11.030651] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 11.075089] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 11.120574] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 11.131741] Non-volatile memory driver v1.3 [ 11.136644] Linux agpgart interface v0.103 [ 11.393259] virtio_blk virtio1: [vda] 139272 512-byte logical blocks (71.3 MB/68.0 MiB) [ 11.440952] vda: detected capacity change from 0 to 71307264 [ 11.690799] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 11.696046] vdb: detected capacity change from 0 to 1073741824 [ 11.907252] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 11.916749] vdc: detected capacity change from 0 to 2621440000 [ 12.077436] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 12.082462] vdd: detected capacity change from 0 to 2621440000 [ 12.442811] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 12.470458] vde: detected capacity change from 0 to 4294967296 [ 12.613363] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 12.626664] vdf: detected capacity change from 0 to 4294967296 [ 12.696218] libphy: Fixed MDIO Bus: probed [ 12.720333] usbcore: registered new interface driver usbserial_generic [ 12.724317] usbserial: USB Serial support registered for generic [ 12.728345] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 12.735247] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 12.737738] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 12.741318] mousedev: PS/2 mouse device common for all mice [ 12.747052] rtc_cmos 00:05: RTC can wake from S4 [ 12.754466] rtc_cmos 00:05: registered as rtc0 [ 12.759302] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 12.766806] intel_pstate: CPU model not supported [ 12.767914] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 12.787068] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 12.797551] hid: raw HID events driver (C) Jiri Kosina [ 12.805734] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 12.839607] usbcore: registered new interface driver usbhid [ 12.850150] usbhid: USB HID core driver [ 12.855692] drop_monitor: Initializing network drop monitor service [ 12.862449] Initializing XFRM netlink socket [ 12.866371] NET: Registered protocol family 10 [ 12.877414] Segment Routing with IPv6 [ 12.881991] NET: Registered protocol family 17 [ 12.886448] mpls_gso: MPLS GSO support [ 12.904152] RAS: Correctable Errors collector initialized. [ 12.913776] AVX version of gcm_enc/dec engaged. [ 12.920177] AES CTR mode by8 optimization enabled [ 13.355590] sched_clock: Marking stable (13355543354, 0)->(16304576172, -2949032818) [ 13.380789] registered taskstats version 1 [ 13.396755] Loading compiled-in X.509 certificates [ 13.465728] zswap: loaded using pool lzo/zbud [ 13.548982] Key type big_key registered [ 13.589707] Key type encrypted registered [ 13.594230] ima: No TPM chip found, activating TPM-bypass! [ 13.599265] ima: Allocated hash algorithm: sha1 [ 13.604518] ima: No architecture policies found [ 13.608683] evm: Initialising EVM extended attributes: [ 13.614280] evm: security.selinux [ 13.616818] evm: security.ima [ 13.619375] evm: security.capability [ 13.621975] evm: HMAC attrs: 0x1 [ 13.641707] rtc_cmos 00:05: setting system clock to 2026-06-18 21:15:18 UTC (1781817318) [ 13.657977] debug: unmapping init [mem 0xffffffffbca03000-0xffffffffbcbfffff] [ 13.664523] debug: unmapping init [mem 0xffffffffbb782000-0xffffffffbba58fff] [ 13.678164] Write protecting the kernel read-only data: 28672k [ 13.685313] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 13.690190] debug: unmapping init [mem 0xffffffffba714000-0xffffffffba7fffff] [ 13.796414] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.814527] systemd[1]: Detected virtualization kvm. [ 13.818651] systemd[1]: Detected architecture x86-64. [ 13.822686] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 13.876397] systemd[1]: No hostname configured. [ 13.880439] systemd[1]: Set hostname to . [ 13.884290] random: systemd: uninitialized urandom read (16 bytes read) [ 13.889776] systemd[1]: Initializing machine ID from random generator. [ 14.063382] random: ln: uninitialized urandom read (6 bytes read) [ 14.313643] random: systemd: uninitialized urandom read (16 bytes read) [ 14.321713] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 14.348034] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 14.411615] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 17.382232] device-mapper: uevent: version 1.0.3 [ 17.392251] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [0[ 20.515388] virtio_net virtio0 ens2: renamed from eth0 m] Reached target Basic System. [ OK [ 20.594306] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 22.568970] scsi host0: ata_piix [ 22.627957] scsi host1: ata_piix [ 22.632790] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 22.640796] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 27.165598] random: crng init done [ 27.169186] random: 7 urandom warning(s) missed due to ratelimiting [ 30.047194] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 33.207319] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 39.507765] printk: systemd: 26 output lines suppressed due to ratelimiting [ 40.934166] SELinux: Disabled at runtime. [ 41.167178] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 41.193512] systemd[1]: Detected virtualization kvm. [ 41.203382] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 43.855114] systemd[1]: initrd-switch-root.service: Succeeded. [ 43.869023] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 43.893475] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 43.909860] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 43.931798] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 43.986345] systemd[1]: Starting Journal Service... Starting Journal Service... [ 44.057367] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ 45.070587] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 46.376341] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 48.239749] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 48.250325] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 49.151106] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 49.287234] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 56.736248] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit)[ 59.263586] NFS: Registering the id_resolver key type [ 59.269761] Key type id_resolver registered [ 59.282455] Key type id_legacy registered [* ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg130-server login: [ 112.725866] libcfs: loading out-of-tree module taints kernel. [ 112.774107] Key type ._llcrypt registered [ 112.777840] Key type .llcrypt registered [ 112.878444] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 127.801556] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 129.270046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 129.288595] alg: No test for adler32 (adler32-zlib) [ 130.597488] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 131.341338] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 133.087197] Key type lgssc registered [ 134.924705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.601704] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.925382] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 201.547334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.573334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.830290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.899989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.015929] Lustre: lustre-MDT0000: new disk, initializing [ 203.112457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.147966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.101781] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.811415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.958392] Lustre: 6507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.041341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.052581] Lustre: Skipped 1 previous similar message [ 220.199567] Lustre: lustre-MDT0001: new disk, initializing [ 220.301818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.343743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.365878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.162410] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.618505] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 237.466244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.778345] Lustre: lustre-OST0000: new disk, initializing [ 237.784817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.786205] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 237.797250] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.881913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.204507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.214239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.270921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.661899] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.083731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.250950] Lustre: lustre-OST0001: new disk, initializing [ 257.257647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 257.270278] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.363709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.263779] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.745220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.753446] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.791737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.755765] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.399430] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.641367] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.532793] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing yml_node [ 292.173249] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 294.311388] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 296.680860] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 298.255275] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 18 17:20:01 EDT 2026 [ 300.311357] hrtimer: interrupt took 4343065 ns [ 313.500330] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 314.944621] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.257164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.262926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.273476] Lustre: Skipped 2 previous similar messages [ 332.292102] Lustre: Skipped 3 previous similar messages [ 336.074419] Lustre: server umount lustre-MDT0000 complete [ 339.607298] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781817644 with bad export cookie 161255739467583620 [ 339.628863] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.641974] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 342.497534] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.499365] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.507768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.515790] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 342.532780] Lustre: Skipped 2 previous similar messages [ 344.545466] LustreError: 7818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.547948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.562728] LustreError: 7818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 344.568638] Lustre: Skipped 2 previous similar messages [ 345.923720] Lustre: server umount lustre-MDT0001 complete [ 349.440339] Lustre: server umount lustre-OST0000 complete [ 353.108653] Lustre: server umount lustre-OST0001 complete [ 358.372280] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 365.763284] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 394.714418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.919495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.952527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.026310] Lustre: lustre-MDT0000: new disk, initializing [ 395.126572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.151705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 398.158138] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 407.023536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.166273] Lustre: 15066:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.260430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.266324] Lustre: Skipped 1 previous similar message [ 407.342233] Lustre: lustre-MDT0001: new disk, initializing [ 407.437249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.474484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.483584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 410.976921] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 419.375534] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 424.319094] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 429.606045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.947911] Lustre: lustre-OST0000: new disk, initializing [ 429.954904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.969289] Lustre: 17036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 430.078119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.023987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 432.042395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 432.127907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.415671] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 443.357789] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 446.911843] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.164145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.048701] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.217457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.705349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.706922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.708252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.719340] Lustre: Skipped 1 previous similar message [ 457.725250] LustreError: Skipped 1 previous similar message [ 458.859631] Lustre: server umount lustre-OST0000 complete [ 462.818704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.938755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.949745] Lustre: Skipped 4 previous similar messages [ 477.151435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 477.372716] Lustre: server umount lustre-MDT0000 complete [ 480.683759] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781817785 with bad export cookie 161255739467585153 [ 480.684528] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.696293] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 480.942695] Lustre: server umount lustre-MDT0001 complete [ 484.661401] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:23:08 (1781817788) [ 486.909401] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 488.587693] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:23:12 (1781817792) [ 495.404238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.945864] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.050783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.215303] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 501.221646] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.237414] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 505.321454] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.096431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.389364] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 519.453928] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.650827] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 529.942204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.714544] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 541.181232] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 550.881525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.882531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.894293] LustreError: Skipped 1 previous similar message [ 550.896743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.919494] Lustre: Skipped 2 previous similar messages [ 550.936078] Lustre: Skipped 4 previous similar messages [ 553.394545] Lustre: server umount lustre-OST0000 complete [ 560.610373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 560.623268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.414235] Lustre: server umount lustre-MDT0000 complete [ 566.252998] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.301997] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 567.500885] LustreError: 21291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781817872 with bad export cookie 161255739467586035 [ 567.505257] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.515361] LustreError: 21291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 567.818326] Lustre: server umount lustre-MDT0001 complete [ 578.452914] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 581.170096] Key type lgssc unregistered [ 581.528960] LNet: 22982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.571988] LNet: Removed LNI 192.168.201.130@tcp [ 583.308190] Key type .llcrypt unregistered [ 583.311966] Key type ._llcrypt unregistered [ 605.955936] Key type ._llcrypt registered [ 605.959513] Key type .llcrypt registered [ 606.100234] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 607.048372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.239432] alg: No test for adler32 (adler32-zlib) [ 608.444819] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 608.656115] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 610.327427] Key type lgssc registered [ 611.778631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.727746] loop: module loaded [ 620.831745] loop0: detected capacity change from 0 to 204800000 [ 621.066284] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.098714] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.439930] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.128372] loop0: detected capacity change from 0 to 204800000 [ 628.381088] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.398990] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.416457] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.372471] loop0: detected capacity change from 0 to 204800000 [ 638.721570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 638.741530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.049981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 640.106304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 640.176994] Lustre: lustre-MDT0000: new disk, initializing [ 640.269954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.296384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 644.421577] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 652.546400] loop1: detected capacity change from 0 to 204800000 [ 657.907563] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 658.125805] Lustre: lustre-OST07c6: new disk, initializing [ 658.129554] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 658.135514] Lustre: Skipped 1 previous similar message [ 658.141944] Lustre: 26170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 658.222480] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 661.704886] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 663.578760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 663.584450] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 663.602690] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 669.865188] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 669.871967] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 669.876884] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 669.884223] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 669.890635] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 669.895706] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 682.700428] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 682.706238] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 682.709935] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 682.713931] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 682.719470] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 682.722591] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.519727] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 686.524987] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.529730] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 686.533122] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.536668] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.540454] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.545031] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 686.550681] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.555710] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.560088] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.565543] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.569515] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.692127] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 689.703124] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 689.709216] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.715763] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.722471] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.727908] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.733649] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 689.739483] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.744474] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 689.750905] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.755516] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.759371] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.627523] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 696.635177] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 696.641617] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 696.648747] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.654498] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.662588] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.667488] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 696.674198] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.680972] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.689080] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.700036] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.709124] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 706.705254] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 706.713686] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 706.719873] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 706.723347] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 706.727423] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.731089] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 706.737168] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 706.743384] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 706.750842] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 706.758016] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 706.763971] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.769360] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.562076] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.563321] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.586561] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 754.931989] Lustre: server umount lustre-OST07c6 complete [ 760.284254] Lustre: server umount lustre-MDT0000 complete [ 770.221739] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 772.422283] Key type lgssc unregistered [ 772.686590] LNet: 28731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.733839] LNet: Removed LNI 192.168.201.130@tcp [ 774.250277] Key type .llcrypt unregistered [ 774.252907] Key type ._llcrypt unregistered [ 785.083621] Key type ._llcrypt registered [ 785.086094] Key type .llcrypt registered [ 785.175330] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 785.693259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 785.782538] alg: No test for adler32 (adler32-zlib) [ 786.721393] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 786.857908] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 788.463139] Key type lgssc registered [ 789.070713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 793.869544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 793.886223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.268090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.766242] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 802.695055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.960762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 805.313377] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 809.864270] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 813.033924] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 816.892944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 817.139617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.206465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 820.692061] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 824.749105] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 837.602082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 837.602455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.604518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 837.615697] Lustre: Skipped 1 previous similar message [ 838.820692] Lustre: server umount lustre-OST0000 complete [ 842.721925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.723506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 842.728804] Lustre: Skipped 1 previous similar message [ 842.736263] Lustre: Skipped 2 previous similar messages [ 846.969817] Lustre: server umount lustre-MDT0000 complete [ 847.840829] LustreError: 29981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.849940] LustreError: 29981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 848.899747] LustreError: 30717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781818153 with bad export cookie 12853082649032853547 [ 848.908777] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.911365] LustreError: 30717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 849.110783] Lustre: server umount lustre-MDT0001 complete [ 855.979427] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 857.445479] Key type lgssc unregistered [ 857.625497] LNet: 33710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.661409] LNet: Removed LNI 192.168.201.130@tcp [ 859.095242] Key type .llcrypt unregistered [ 859.096884] Key type ._llcrypt unregistered [ 868.866566] Key type ._llcrypt registered [ 868.869037] Key type .llcrypt registered [ 868.937533] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 869.674670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 869.708371] alg: No test for adler32 (adler32-zlib) [ 870.657695] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 870.781438] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 872.391153] Key type lgssc registered [ 873.029981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 877.391111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 877.413726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.711067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.868045] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 884.699731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.898518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 886.942603] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 890.539877] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 893.326660] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.860389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.030039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.144182] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 903.212931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 903.809544] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 913.376179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.380347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.386665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 914.403177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 914.403927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 914.409224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.428700] Lustre: server umount lustre-OST0000 complete [ 919.521370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.523149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.531489] Lustre: Skipped 1 previous similar message [ 919.537368] Lustre: Skipped 1 previous similar message [ 923.583668] Lustre: server umount lustre-MDT0000 complete [ 924.642977] LustreError: 36119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.657194] LustreError: 36119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 925.523336] LustreError: 34937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781818230 with bad export cookie 17077908039659078987 [ 925.528316] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 925.532861] LustreError: 34937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 925.770634] Lustre: server umount lustre-MDT0001 complete [ 932.359542] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 934.044916] Key type lgssc unregistered [ 934.289569] LNet: 38491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.333403] LNet: Removed LNI 192.168.201.130@tcp [ 935.799591] Key type .llcrypt unregistered [ 935.802315] Key type ._llcrypt unregistered [ 948.281938] Key type ._llcrypt registered [ 948.284671] Key type .llcrypt registered [ 948.350094] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 949.083994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.099060] alg: No test for adler32 (adler32-zlib) [ 950.012239] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 950.123703] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 951.735183] Key type lgssc registered [ 952.354606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 956.623801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.632765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.888482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.054167] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 964.504491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.705713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.957606] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 971.023706] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 973.955949] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 977.255432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 977.441338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.759568] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 982.585092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 985.743272] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 998.496534] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 999.911436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.918465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.932959] Lustre: Skipped 2 previous similar messages [ 999.944638] Lustre: Skipped 2 previous similar messages [ 1001.972875] Lustre: server umount lustre-MDT0000 complete [ 1003.581267] LustreError: 39882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1003.602696] LustreError: 39882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1005.028898] LustreError: 41435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.075064] LustreError: 41435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1006.887615] LustreError: 39866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781818311 with bad export cookie 7633110967111167684 [ 1006.912525] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1006.915569] LustreError: 39866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1008.708634] LustreError: 39880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.744253] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [ 1010.150549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.170964] LustreError: 39881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.171101] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1011.192478] LustreError: 39881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1011.205609] Lustre: Skipped 1 previous similar message [ 1013.245982] Lustre: server umount lustre-MDT0001 complete [ 1032.227706] Lustre: server umount lustre-OST0000 complete [ 1067.643677] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1072.427753] Key type lgssc unregistered [ 1072.929648] LNet: 43769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.061820] LNet: Removed LNI 192.168.201.130@tcp [ 1075.636482] Key type .llcrypt unregistered [ 1075.644238] Key type ._llcrypt unregistered [ 1113.217681] Key type ._llcrypt registered [ 1113.222310] Key type .llcrypt registered [ 1113.372298] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1114.594154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1114.777332] alg: No test for adler32 (adler32-zlib) [ 1116.126631] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 1116.471873] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1118.247239] Key type lgssc registered [ 1120.229162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.259722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1130.283535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.989406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.378842] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1141.226902] LustreError: 45191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.245991] LustreError: 45191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1146.360720] LustreError: 45192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.625251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.238346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1154.735569] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1163.697405] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1169.990636] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1178.007816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.655172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.804637] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1194.544870] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1201.218834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1209.321960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1209.333556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.362802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.365379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.373442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.385518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1211.389768] Lustre: Skipped 1 previous similar message [ 1212.668989] Lustre: server umount lustre-OST0000 complete [ 1236.397809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.419927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.436638] Lustre: Skipped 2 previous similar messages [ 1238.702206] Lustre: server umount lustre-MDT0000 complete [ 1241.570571] LustreError: 46161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.593089] LustreError: 46161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1243.892195] LustreError: 45172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781818548 with bad export cookie 13796364904994794231 [ 1243.899212] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.910340] LustreError: 45172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1250.397815] Lustre: server umount lustre-MDT0001 complete [ 1264.420130] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1268.088524] Key type lgssc unregistered [ 1268.474316] LNet: 48907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.558281] LNet: Removed LNI 192.168.201.130@tcp [ 1270.802405] Key type .llcrypt unregistered [ 1270.806143] Key type ._llcrypt unregistered [ 1322.130577] Key type ._llcrypt registered [ 1322.146327] Key type .llcrypt registered [ 1322.781443] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1327.887514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1329.814537] alg: No test for adler32 (adler32-zlib) [ 1332.912663] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 1334.057193] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1336.567384] Key type lgssc registered [ 1342.835515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.849757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.937768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.342616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.522379] LustreError: 50336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.622232] LustreError: 50336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1392.148480] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1393.639855] LustreError: 50337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.765658] LustreError: 50336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.880530] LustreError: 50337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.999630] LustreError: 50336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.249405] LustreError: 50337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.342115] LustreError: 50337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1420.513249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.286307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.563918] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1456.486134] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1471.708248] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1495.070980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1496.045653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.104923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1512.775728] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1532.255099] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1548.771819] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1558.984278] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1575.148494] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1817h41m14s [ 1579.522698] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1586.153588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1586.171091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.200322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.307154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.350457] Lustre: Skipped 4 previous similar messages [ 1588.424227] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 1590.691387] Lustre: server umount lustre-MDT0000 complete [ 1593.855819] LustreError: 50332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.885175] LustreError: 50332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1598.804136] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1608.159274] Lustre: 49645:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781818897/real 1781818897] req@0000000031c3ec44 x1868372260228480/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781818913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1608.268811] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.252796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1618.602189] LustreError: 49641:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001824c28c x1868372260236160/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1619.493923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.717986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1620.543564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1630.824299] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1639.071915] Lustre: DEBUG MARKER: Wait for df (36506) ... [ 1642.475612] Lustre: DEBUG MARKER: done [ 1654.756226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1654.791424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.862419] Lustre: Skipped 2 previous similar messages [ 1654.883773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1660.059552] Lustre: server umount lustre-OST0000 complete [ 1671.653609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.670789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.680691] Lustre: Skipped 2 previous similar messages [ 1671.701487] Lustre: Skipped 4 previous similar messages [ 1677.552812] Lustre: server umount lustre-MDT0000 complete [ 1678.821445] LustreError: 51068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.863221] LustreError: 51068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 1687.794062] LustreError: 53002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781818992 with bad export cookie 3881235779632947100 [ 1687.839977] LustreError: 53002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1687.845930] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.972664] Lustre: server umount lustre-MDT0001 complete [ 1725.705533] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1734.276782] Key type lgssc unregistered [ 1735.053176] LNet: 55335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.139524] LNet: Removed LNI 192.168.201.130@tcp [ 1738.608356] Key type .llcrypt unregistered [ 1738.641439] Key type ._llcrypt unregistered [ 1851.496117] Key type ._llcrypt registered [ 1851.513302] Key type .llcrypt registered [ 1852.124604] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1855.010423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.394885] alg: No test for adler32 (adler32-zlib) [ 1858.190417] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 1858.799033] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1860.688241] Key type lgssc registered [ 1864.168594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1895.502042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1895.692553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.671268] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1898.806670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1900.220444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.969790] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1937.856899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.209122] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1938.364059] Lustre: 57558:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1939.068202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.057572] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1978.660418] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1994.150440] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2013.373187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2013.915874] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2014.775196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.318084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2036.180400] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2057.770450] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2072.413153] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1817h49m33s [ 2076.608815] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2096.781946] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2103.684149] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2103.758329] LustreError: 58722:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2124.054166] Lustre: lustre-MDT0000: Client 365dee6f-7cd5-4dd9-b946-439cdcd13cdd (at 192.168.201.30@tcp) reconnecting [ 2143.823233] LustreError: 58722:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2150.813550] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2157.141160] Lustre: DEBUG MARKER: done [ 2173.967743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2173.970603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.977656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2174.008180] LustreError: Skipped 1 previous similar message [ 2174.132108] Lustre: Skipped 1 previous similar message [ 2179.067380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.091478] Lustre: Skipped 2 previous similar messages [ 2179.829634] Lustre: server umount lustre-OST0000 complete [ 2189.281190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.332582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.358898] Lustre: Skipped 1 previous similar message [ 2195.886978] Lustre: server umount lustre-MDT0000 complete [ 2203.113198] LustreError: 58427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.224134] LustreError: 58427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2205.230264] LustreError: 57568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.288137] LustreError: 57568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2207.063624] LustreError: 59434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781819511 with bad export cookie 14961169558205295417 [ 2207.080659] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.115923] LustreError: 59434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2208.423711] Lustre: server umount lustre-MDT0001 complete [ 2240.577933] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2248.250582] Key type lgssc unregistered [ 2249.270650] LNet: 61940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.367584] LNet: Removed LNI 192.168.201.130@tcp [ 2252.644687] Key type .llcrypt unregistered [ 2252.652430] Key type ._llcrypt unregistered [ 2360.175848] Key type ._llcrypt registered [ 2360.182413] Key type .llcrypt registered [ 2360.528202] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2362.334934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.473103] alg: No test for adler32 (adler32-zlib) [ 2364.073956] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 2364.542935] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2366.279187] Key type lgssc registered [ 2368.587714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.814662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.845663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.486709] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2377.560539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2378.050796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.778338] Lustre: Failing over lustre-MDT0000 [ 2380.361634] Lustre: server umount lustre-MDT0000 complete [ 2393.348344] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:54:55 (1781819695) [ 2405.852486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.704836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.369138] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2431.297720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.444413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2431.485345] Lustre: 64762:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2431.920841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.307279] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2454.002916] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2465.445698] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2478.966382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.328757] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2479.796279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.536139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2491.144629] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2505.620173] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2514.207365] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2526.085876] Lustre: server umount lustre-MDT0000 complete [ 2528.749937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.776137] Lustre: Skipped 1 previous similar message [ 2532.067788] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781819836 with bad export cookie 870332493302340146 [ 2532.071038] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.090556] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2532.594740] Lustre: server umount lustre-MDT0001 complete [ 2539.068962] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2549.218346] Lustre: 63685:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819838/real 1781819838] req@00000000f52300ab x1868373341291392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781819854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2549.258736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.285405] Lustre: Skipped 1 previous similar message [ 2549.298452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2553.247373] Lustre: 63682:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819842/real 1781819842] req@000000002028aa79 x1868373341291520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781819858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2554.336188] Lustre: 63682:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819843/real 1781819843] req@00000000e8b1b257 x1868373341291648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781819859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2557.471350] Lustre: 63685:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819846/real 1781819846] req@000000002b5e5cbd x1868373341291904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781819862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2562.348745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.680333] Lustre: 63685:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819852/real 1781819852] req@000000002e041990 x1868373341292032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781819868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2576.928917] LustreError: 63681:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005452dfb2 x1868373341293056/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2577.437895] LustreError: 66617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.502508] LustreError: 66617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2577.640544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.864656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2579.619775] LustreError: 66618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.069781] LustreError: 66617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.578378] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2590.183512] LustreError: 66618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.288957] LustreError: 68550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.308348] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2594.319420] LustreError: 68550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2600.612940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.452422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2606.576315] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2607.819445] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2619.177222] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2627.203786] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2637.300505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.314564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.340890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.404438] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.406397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.418359] Lustre: Skipped 1 previous similar message [ 2642.424242] Lustre: Skipped 1 previous similar message [ 2643.665852] Lustre: server umount lustre-MDT0000 complete [ 2647.528113] LustreError: 69242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.559161] LustreError: 69242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2649.543485] LustreError: 66625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781819954 with bad export cookie 870332493302342421 [ 2649.559623] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2649.574289] LustreError: 66625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2650.105814] Lustre: server umount lustre-MDT0001 complete [ 2659.661994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2669.023253] Lustre: 63682:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819957/real 1781819957] req@000000009f7553bd x1868373341323776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781819973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2669.097888] Lustre: 63682:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2669.109830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.222809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.754279] LustreError: 63681:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000057d9d721 x1868373341325184/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2695.458362] LustreError: 66618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.586353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.725388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2697.672260] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2702.817740] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2716.643480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.808386] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2725.230808] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2740.375030] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2750.389526] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2763.234280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2763.265775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.304851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.868200] LustreError: 71238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.882626] LustreError: 71238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2768.914631] Lustre: server umount lustre-MDT0000 complete [ 2773.859452] LustreError: 64754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781820078 with bad export cookie 870332493302343800 [ 2773.869106] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.879240] LustreError: 64754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2773.997376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2774.008675] Lustre: Skipped 2 previous similar messages [ 2774.265895] Lustre: server umount lustre-MDT0001 complete [ 2783.489671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2788.815905] Lustre: DEBUG MARKER: files compared the same [ 2792.340339] Lustre: server umount lustre-OST0000 complete [ 2821.524702] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2826.740447] Key type lgssc unregistered [ 2827.285973] LNet: 74552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.339461] LNet: Removed LNI 192.168.201.130@tcp [ 2829.752165] Key type .llcrypt unregistered [ 2829.756082] Key type ._llcrypt unregistered [ 2862.685921] Key type ._llcrypt registered [ 2862.692919] Key type .llcrypt registered [ 2862.939175] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2864.525283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.590830] alg: No test for adler32 (adler32-zlib) [ 2865.951235] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 2866.553295] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2868.463435] Key type lgssc registered [ 2870.404391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2883.487394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2883.522916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.363265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.966109] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2895.847856] LustreError: 75971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.875058] LustreError: 75971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2900.969139] LustreError: 75970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.311487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.789687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.167338] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2921.726780] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2928.681247] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2935.902449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.336492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.762616] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2950.207080] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2955.816902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2965.996855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2966.004887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.020202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.042050] Lustre: Skipped 1 previous similar message [ 2969.337637] Lustre: server umount lustre-OST0000 complete [ 2976.225861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.228545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.250073] Lustre: Skipped 1 previous similar message [ 2976.253714] Lustre: Skipped 1 previous similar message [ 2980.449298] Lustre: server umount lustre-MDT0000 complete [ 2981.350232] LustreError: 75970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.394409] LustreError: 75970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2985.446660] LustreError: 75971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.490455] LustreError: 75971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2985.512640] LustreError: 75952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781820290 with bad export cookie 11169625781820817197 [ 2985.532231] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.536840] LustreError: 75952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2985.944037] Lustre: server umount lustre-MDT0001 complete [ 2999.095531] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3002.339470] Key type lgssc unregistered [ 3002.861445] LNet: 79524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3003.879927] LNet: Removed LNI 192.168.201.130@tcp [ 3005.134408] Key type .llcrypt unregistered [ 3005.146489] Key type ._llcrypt unregistered [ 3043.842568] Key type ._llcrypt registered [ 3043.847788] Key type .llcrypt registered [ 3044.064530] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3045.876272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3045.961496] alg: No test for adler32 (adler32-zlib) [ 3047.580609] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 3048.385180] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3050.189479] Key type lgssc registered [ 3052.460855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3066.481216] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3066.499861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.213045] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3087.327476] LustreError: 80920:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.130@tcp: failed processing log, type 4: rc = -110 [ 3114.082115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.851455] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3136.186571] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3146.832637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.069554] Lustre: *** cfs_fail_loc=706, val=0*** [ 3147.611477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.717398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3152.708704] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3153.903825] LustreError: 82007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.925139] LustreError: 82007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3159.012787] LustreError: 80946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.062909] LustreError: 80946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3163.134251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.891499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3172.245035] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3190.569686] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3201.081490] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3209.189150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3209.209113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.244426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.214086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.217463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.294712] Lustre: Skipped 1 previous similar message [ 3211.564925] Lustre: server umount lustre-OST0000 complete [ 3219.937066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.961940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.996808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.573474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.599944] Lustre: Skipped 3 previous similar messages [ 3230.689696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.706180] Lustre: Skipped 1 previous similar message [ 3232.223926] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3232.616476] Lustre: server umount lustre-MDT0000 complete [ 3236.836894] LustreError: 82013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.889731] LustreError: 82013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3239.484681] LustreError: 82751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781820544 with bad export cookie 15751967497233153828 [ 3239.505603] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.521678] LustreError: 82751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3240.091393] Lustre: server umount lustre-MDT0001 complete [ 3257.039854] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3262.833898] Key type lgssc unregistered [ 3263.299293] LNet: 84547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3264.358615] LNet: Removed LNI 192.168.201.130@tcp [ 3265.879502] Key type .llcrypt unregistered [ 3265.900551] Key type ._llcrypt unregistered [ 3298.826517] Key type ._llcrypt registered [ 3298.831773] Key type .llcrypt registered [ 3299.067588] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3301.164522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3301.235346] alg: No test for adler32 (adler32-zlib) [ 3302.430689] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 3302.943582] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3304.815313] Key type lgssc registered [ 3306.786419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3317.858992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3317.892073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.426623] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3338.278925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.068361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3345.380208] LustreError: 86654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.421588] LustreError: 86654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3349.347551] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3349.482324] LustreError: 86657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.534547] LustreError: 86657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3354.596894] LustreError: 86654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.635881] LustreError: 86654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3359.718674] LustreError: 86658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.755090] LustreError: 86658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3364.529547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.837748] LustreError: 86654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.882779] LustreError: 86654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3365.187170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.471981] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3375.595743] LustreError: 86657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.631070] LustreError: 86657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3387.995661] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3388.031794] Lustre: MGS: Client eb1fbe0e-0e32-4f4c-b005-8863e3206472 (at 0@lo) reconnecting [ 3388.068649] LustreError: 86668:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3388.352537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.452701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3395.247559] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3419.108866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.115176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.159542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.173166] Lustre: Skipped 1 previous similar message [ 3419.349630] Lustre: server umount lustre-OST0000 complete [ 3429.346472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3429.350695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.372649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.399126] Lustre: Skipped 2 previous similar messages [ 3433.964801] Lustre: server umount lustre-MDT0000 complete [ 3434.477298] LustreError: 86657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.519545] LustreError: 86657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3440.665780] Lustre: server umount lustre-MDT0001 complete [ 3445.659556] Lustre: server umount MGS complete [ 3465.331977] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:12:46 (1781820766) [ 3514.766779] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 3531.758711] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3587.245849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.924454] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3608.330315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.647705] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3608.724594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3608.839913] Lustre: lustre-MDT0001: new disk, initializing [ 3608.991778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.036981] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3609.043703] Lustre: 93892:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3609.063946] LustreError: 93892:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3616.377733] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3634.047218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.521388] Lustre: lustre-OST0000: new disk, initializing [ 3634.526246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3634.541218] Lustre: 94661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3634.645422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.021766] Lustre: 94842:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000198fa3ce x1868374324854272/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781820962 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3647.232643] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3662.817657] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3664.660589] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3675.930073] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3675.951643] Lustre: MGS: Client 81a34d15-0850-4247-a989-b4c58d6a641a (at 0@lo) reconnecting [ 3676.027922] Lustre: 93156:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3676.053093] LustreError: 93156:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3676.061944] LustreError: 93156:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3676.101789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3676.253716] Lustre: lustre-MDT0000: new disk, initializing [ 3676.510136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.586306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3677.681141] Lustre: 95733:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d17fff48 x1868374324862464/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781820998 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3679.909875] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3682.872684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3682.919941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3683.059624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3683.669601] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3717.622535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.635821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.659088] Lustre: Skipped 1 previous similar message [ 3717.678666] Lustre: Skipped 2 previous similar messages [ 3721.983290] Lustre: server umount lustre-OST0000 complete [ 3732.964173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3732.970568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.003866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3733.044065] Lustre: Skipped 1 previous similar message [ 3733.079148] Lustre: Skipped 1 previous similar message [ 3737.188721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.203284] Lustre: Skipped 1 previous similar message [ 3737.469382] Lustre: server umount lustre-MDT0000 complete [ 3742.691307] LustreError: 93864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.748147] LustreError: 93864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3749.752939] Lustre: server umount lustre-MDT0001 complete [ 3755.263864] Lustre: server umount MGS complete [ 3773.156153] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:17:55 (1781821075) [ 3804.097097] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3809.869292] Key type lgssc unregistered [ 3810.448268] LNet: 98238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3811.523785] LNet: Removed LNI 192.168.201.130@tcp [ 3813.084840] Key type .llcrypt unregistered [ 3813.089292] Key type ._llcrypt unregistered [ 3839.674125] Key type ._llcrypt registered [ 3839.677443] Key type .llcrypt registered [ 3839.815939] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3841.326891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3841.491133] alg: No test for adler32 (adler32-zlib) [ 3842.849291] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 3843.130711] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3844.863393] Key type lgssc registered [ 3846.829109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3860.089656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.159295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.132603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.252755] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3871.206119] LustreError: 99464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.235135] LustreError: 99464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3876.331263] LustreError: 99465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.450174] LustreError: 99464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.295971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.841808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3890.806405] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3903.488281] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3912.748986] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3923.020398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3923.626850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.916684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3934.065947] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3948.261598] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3964.908281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3964.927322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.957246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3967.972188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3967.982236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3968.001170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.039970] Lustre: Skipped 1 previous similar message [ 3973.090295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.122324] Lustre: Skipped 1 previous similar message [ 3978.210301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3978.222752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3978.254251] Lustre: Skipped 1 previous similar message [ 3978.469274] Lustre: server umount lustre-OST0000 complete [ 3986.409064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3986.441950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.481974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.455293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.920958] Lustre: server umount lustre-MDT0000 complete [ 3993.571721] LustreError: 99464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.597221] LustreError: 99464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3996.673844] LustreError: 99445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781821301 with bad export cookie 6399008178438987235 [ 3996.675535] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.695203] LustreError: 99445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3997.282804] Lustre: server umount lustre-MDT0001 complete [ 4012.937604] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4017.553785] Key type lgssc unregistered [ 4018.143863] LNet: 103026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4019.208648] LNet: Removed LNI 192.168.201.130@tcp [ 4020.464738] Key type .llcrypt unregistered [ 4020.470113] Key type ._llcrypt unregistered [ 4044.575133] Key type ._llcrypt registered [ 4044.589491] Key type .llcrypt registered [ 4044.879400] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4046.391768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4046.493347] alg: No test for adler32 (adler32-zlib) [ 4048.197830] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 4048.570686] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4050.351955] Key type lgssc registered [ 4052.331813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4062.433850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4062.449131] LustreError: 104112:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 4062.482214] LustreError: 104113:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 4062.495629] LustreError: 104112:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 4062.509939] LustreError: 104112:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4062.525242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.530444] LustreError: 104112:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4064.622558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.979828] LustreError: 104150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.017559] LustreError: 104150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4075.942704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.533494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.112085] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4091.129276] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing lsmod [ 4096.220963] LustreError: 105307:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 4096.427393] LustreError: 105322:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 4096.439221] LustreError: 105307:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4793 [ 4096.456987] LustreError: 105307:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4096.469155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.473077] LustreError: 105307:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4097.283133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.730937] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.743697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.757574] Lustre: Skipped 2 previous similar messages [ 4106.777528] Lustre: Skipped 1 previous similar message [ 4109.702386] Lustre: server umount lustre-MDT0000 complete [ 4111.840957] LustreError: 104151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.912910] LustreError: 104151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4116.962604] LustreError: 104151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.010092] LustreError: 104151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4117.047744] LustreError: 104130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781821421 with bad export cookie 3209600953282077357 [ 4117.065812] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.074136] LustreError: 104130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4119.073902] LustreError: 104584:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bfa6a956 x1868375107067392/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4119.402877] Lustre: server umount lustre-MDT0001 complete [ 4135.618354] Lustre: server umount lustre-OST0000 complete [ 4151.354741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.121370] LustreError: 106391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.350264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.409813] LustreError: 106392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.901291] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4167.659476] LustreError: 106392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.696278] LustreError: 106392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4175.262543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.067373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4183.764900] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4197.172852] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4207.533698] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4218.857104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4219.456461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.538076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 4228.880926] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4241.630345] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4251.111150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4251.137513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.183118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.160711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.164514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.193135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4253.230283] Lustre: Skipped 1 previous similar message [ 4257.220920] Lustre: server umount lustre-OST0000 complete [ 4264.182663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.223891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.240522] Lustre: Skipped 1 previous similar message [ 4269.538782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.560061] Lustre: Skipped 1 previous similar message [ 4270.740624] Lustre: server umount lustre-MDT0000 complete [ 4274.660026] LustreError: 106392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.691139] LustreError: 106392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4276.974813] LustreError: 107127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781821581 with bad export cookie 3209600953282078057 [ 4276.990479] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.001116] LustreError: 107127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4277.430074] Lustre: server umount lustre-MDT0001 complete [ 4296.779381] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4301.634190] Key type lgssc unregistered [ 4302.145395] LNet: 109947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4303.210925] LNet: Removed LNI 192.168.201.130@tcp [ 4304.755318] Key type .llcrypt unregistered [ 4304.759981] Key type ._llcrypt unregistered [ 4340.304103] Key type ._llcrypt registered [ 4340.316948] Key type .llcrypt registered [ 4340.533646] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4342.379075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4342.545646] alg: No test for adler32 (adler32-zlib) [ 4343.859339] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 4344.255071] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4346.017614] Key type lgssc registered [ 4348.344828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4362.514220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4362.559487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.349861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.796385] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4374.501154] LustreError: 111367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.540353] LustreError: 111367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4379.622894] LustreError: 111368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.941034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.690539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4389.629893] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4400.661128] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4408.019963] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4418.336039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4418.954783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.700898] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4434.306510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 4441.598329] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4447.632213] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4454.205089] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4465.129942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4465.133236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.142325] LustreError: Skipped 1 previous similar message [ 4465.144189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4465.160718] Lustre: Skipped 1 previous similar message [ 4466.776570] Lustre: server umount lustre-OST0000 complete [ 4475.369410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.376467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.394049] Lustre: Skipped 1 previous similar message [ 4475.427656] Lustre: Skipped 2 previous similar messages [ 4479.291725] Lustre: server umount lustre-MDT0000 complete [ 4480.482038] LustreError: 112312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.529836] LustreError: 112312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4484.568830] LustreError: 111347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781821789 with bad export cookie 2145128749756807997 [ 4484.581214] LustreError: 111347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4484.587539] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4485.045145] Lustre: server umount lustre-MDT0001 complete [ 4500.882687] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4504.675836] Key type lgssc unregistered [ 4504.999872] LNet: 115014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4506.031422] LNet: Removed LNI 192.168.201.130@tcp [ 4507.186488] Key type .llcrypt unregistered [ 4507.191333] Key type ._llcrypt unregistered [ 4532.220845] Key type ._llcrypt registered [ 4532.224594] Key type .llcrypt registered [ 4532.350588] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4533.631192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4533.684223] alg: No test for adler32 (adler32-zlib) [ 4534.847192] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 4535.132247] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4536.879214] Key type lgssc registered [ 4538.736187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4551.133695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4551.167043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.227056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.889865] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4562.408986] LustreError: 116239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.460322] LustreError: 116239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4567.522666] LustreError: 116238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4572.650233] LustreError: 116239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.805190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.543331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.218343] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4594.941653] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4603.736134] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4613.819539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.509808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.737980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4624.459164] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4636.485309] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4648.429085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.444123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.460958] Lustre: Skipped 1 previous similar message [ 4648.488594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4653.418696] Lustre: server umount lustre-OST0000 complete [ 4662.752679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4662.772132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.802374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4662.807250] Lustre: Skipped 1 previous similar message [ 4663.783095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.810556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4663.823231] Lustre: Skipped 1 previous similar message [ 4665.365190] Lustre: server umount lustre-MDT0000 complete [ 4668.901183] LustreError: 116238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.946722] LustreError: 116238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4670.882174] LustreError: 116218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781821975 with bad export cookie 2898260880212675376 [ 4670.886255] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.899333] LustreError: 116218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4671.309215] Lustre: server umount lustre-MDT0001 complete [ 4687.467500] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4691.939171] Key type lgssc unregistered [ 4692.421330] LNet: 119782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4693.484703] LNet: Removed LNI 192.168.201.130@tcp [ 4695.003917] Key type .llcrypt unregistered [ 4695.016110] Key type ._llcrypt unregistered [ 4730.341862] Key type ._llcrypt registered [ 4730.355624] Key type .llcrypt registered [ 4730.572912] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4732.006921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.178788] alg: No test for adler32 (adler32-zlib) [ 4733.420229] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 4733.871656] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4735.584413] Key type lgssc registered [ 4737.572322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4748.927287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4749.052592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.046646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.523421] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4760.559285] LustreError: 121202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.596048] LustreError: 121202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4765.672123] LustreError: 121201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.771090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.786027] LustreError: 121202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.505278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4777.756255] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4786.405761] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4793.976108] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4802.124099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4802.744204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.872395] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4820.262336] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4821.191355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4832.939939] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4839.131250] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4839.147634] Lustre: Skipped 2 previous similar messages [ 4842.608188] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4842.668079] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4845.517279] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4845.533709] Lustre: Skipped 2 previous similar messages [ 4851.349800] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4851.377016] Lustre: Skipped 1 previous similar message [ 4860.175250] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4860.189885] Lustre: Skipped 2 previous similar messages [ 4866.733832] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.30@tcp 192.168.0.[2,10]@tcp [ 4875.162548] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.30/24@tcp in log lustre-MDT0000 [ 4875.185088] Lustre: Skipped 2 previous similar messages [ 4882.563288] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.30/24@tcp [ 4882.585420] Lustre: Skipped 1 previous similar message [ 4893.157653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.163688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.164187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4893.180348] Lustre: Skipped 1 previous similar message [ 4893.197289] Lustre: Skipped 1 previous similar message [ 4896.563745] Lustre: server umount lustre-OST0000 complete [ 4903.405808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.414577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4903.436786] Lustre: Skipped 1 previous similar message [ 4903.465385] Lustre: Skipped 1 previous similar message [ 4908.514331] LustreError: 121198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.542050] LustreError: 121198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4908.591780] Lustre: server umount lustre-MDT0000 complete [ 4913.637621] LustreError: 124301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.667640] LustreError: 124301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4913.949982] LustreError: 121182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781822218 with bad export cookie 6353720685139621702 [ 4913.965153] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4913.982177] LustreError: 121182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4914.434960] Lustre: server umount lustre-MDT0001 complete [ 4929.504091] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4934.601937] Key type lgssc unregistered [ 4935.086476] LNet: 126200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4936.185251] LNet: Removed LNI 192.168.201.130@tcp [ 4937.633490] Key type .llcrypt unregistered [ 4937.637444] Key type ._llcrypt unregistered [ 4977.723047] Key type ._llcrypt registered [ 4977.727211] Key type .llcrypt registered [ 4977.946472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4979.480663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4979.509073] alg: No test for adler32 (adler32-zlib) [ 4980.748329] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 4981.117218] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4982.887347] Key type lgssc registered [ 4984.380296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4994.994597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4995.065343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.707715] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4996.744526] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.30/24@tcp [ 4996.754836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.723856] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5006.305411] LustreError: 127767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.323389] LustreError: 127767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5011.454592] LustreError: 127766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.644186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.141370] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5014.169641] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.30/24@tcp [ 5014.287834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5020.702224] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5033.306423] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5042.014816] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5052.666687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.249516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.363423] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5067.603481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 5074.076291] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5088.236963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5088.237336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.254152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5088.273629] Lustre: Skipped 1 previous similar message [ 5093.349724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5093.360094] Lustre: Skipped 1 previous similar message [ 5094.400119] Lustre: server umount lustre-OST0000 complete [ 5101.538346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5101.560255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.584258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.590620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.596770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.626713] Lustre: Skipped 1 previous similar message [ 5107.118774] Lustre: server umount lustre-MDT0000 complete [ 5108.708218] LustreError: 127766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.734349] LustreError: 127766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5112.480625] LustreError: 130407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781822417 with bad export cookie 10941061428419233916 [ 5112.491915] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5112.509579] LustreError: 130407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5112.945312] Lustre: server umount lustre-MDT0001 complete [ 5130.639172] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5136.095634] Key type lgssc unregistered [ 5136.616891] LNet: 131361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5137.706393] LNet: Removed LNI 192.168.201.130@tcp [ 5139.204913] Key type .llcrypt unregistered [ 5139.207930] Key type ._llcrypt unregistered [ 5176.429627] Key type ._llcrypt registered [ 5176.436470] Key type .llcrypt registered [ 5176.667801] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5195.318801] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5197.098418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5197.171439] alg: No test for adler32 (adler32-zlib) [ 5198.372618] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 5198.789499] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 5200.631255] Key type lgssc registered [ 5202.361232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5253.449472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5253.489820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.782507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5254.821595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5254.902272] Lustre: lustre-MDT0000: new disk, initializing [ 5255.029592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.056492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5260.480275] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5279.849432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.168226] Lustre: 134766:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5280.423210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5280.433762] Lustre: Skipped 1 previous similar message [ 5280.551380] Lustre: lustre-MDT0001: new disk, initializing [ 5280.798167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5280.903544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5280.919696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5287.119579] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5300.808916] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5309.474089] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5317.931455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5318.353548] Lustre: lustre-OST0000: new disk, initializing [ 5318.358167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5318.361873] Lustre: 136736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5318.497725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.973548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5325.004772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5325.195743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5326.206348] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5339.771175] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.911080] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5347.253973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5353.004758] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.293372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5356.007137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5356.011259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.018388] LustreError: Skipped 1 previous similar message [ 5356.027152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5356.038580] Lustre: Skipped 1 previous similar message [ 5361.129338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.146121] Lustre: Skipped 1 previous similar message [ 5366.245962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5370.335826] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5370.629971] Lustre: server umount lustre-OST0000 complete [ 5378.020249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.036571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.050362] Lustre: Skipped 1 previous similar message [ 5378.056961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.060513] Lustre: Skipped 1 previous similar message [ 5381.606487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.850128] Lustre: server umount lustre-MDT0000 complete [ 5388.354758] LustreError: 134757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781822693 with bad export cookie 13561986629829314768 [ 5388.378544] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5388.784444] Lustre: server umount lustre-MDT0001 complete [ 5400.423914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.569783] LustreError: 138960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.618180] LustreError: 138960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5401.730869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.204439] LustreError: 138961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.528920] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5411.301510] LustreError: 138960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.422332] LustreError: 138961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.713975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.437308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5425.291545] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5435.622958] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5445.023330] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5455.995158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.567645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.277309] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5477.485355] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5484.748458] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5485.141370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5492.333829] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5492.752584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5506.228194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.467836] Lustre: lustre-OST0001: new disk, initializing [ 5506.474332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5506.489417] Lustre: 142338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5506.626569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5513.880960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5513.906840] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5514.076650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5515.790970] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5528.843680] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5529.442486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5536.714482] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5537.200302] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5549.403853] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff933199059000.ost_server_uuid 50 [ 5553.103478] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff933199059000.ost_server_uuid in IDLE state after 0 sec [ 5575.677898] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.684053] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5575.684434] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.707925] Lustre: Skipped 1 previous similar message [ 5575.765669] Lustre: Skipped 3 previous similar messages [ 5585.375187] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5585.554665] Lustre: server umount lustre-OST0001 complete [ 5585.894621] LustreError: 140785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.948222] LustreError: 140785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5591.024288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.054208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.079317] Lustre: Skipped 3 previous similar messages [ 5596.148266] LustreError: 142612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.217486] LustreError: 142612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5605.343402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5605.775682] Lustre: server umount lustre-OST0000 complete [ 5616.626882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.659069] Lustre: Skipped 1 previous similar message [ 5616.672229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5616.697625] Lustre: Skipped 5 previous similar messages [ 5620.187556] Lustre: server umount lustre-MDT0000 complete [ 5621.731212] LustreError: 140204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.758170] LustreError: 140204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5625.613532] LustreError: 138941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781822930 with bad export cookie 13561986629829315650 [ 5625.627516] LustreError: 138941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5625.633326] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.190481] Lustre: server umount lustre-MDT0001 complete [ 5659.189225] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5663.762134] Key type lgssc unregistered [ 5664.232711] LNet: 145383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5665.328178] LNet: Removed LNI 192.168.201.130@tcp [ 5666.508650] Key type .llcrypt unregistered [ 5666.511990] Key type ._llcrypt unregistered [ 5729.798974] Key type ._llcrypt registered [ 5729.806404] Key type .llcrypt registered [ 5730.054372] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5753.363951] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5755.285808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5755.481757] alg: No test for adler32 (adler32-zlib) [ 5756.934707] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 5757.304715] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 5759.041565] Key type lgssc registered [ 5761.162544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5846.097708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5846.162475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.794708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5847.866358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5847.960643] Lustre: lustre-MDT0000: new disk, initializing [ 5848.167559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.212622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5858.264242] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5878.944876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.283761] Lustre: 149554:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5879.500586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5879.519214] Lustre: Skipped 1 previous similar message [ 5879.791430] Lustre: lustre-MDT0001: new disk, initializing [ 5880.113180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5880.245359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5880.283723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5890.782660] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5910.798269] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5925.540581] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5944.378515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5945.197455] Lustre: lustre-OST0000: new disk, initializing [ 5945.203646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5945.226622] Lustre: 151532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5945.425136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.766250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5952.805297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5953.150412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5961.540791] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5980.622085] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5988.190802] Lustre: Failing over lustre-OST0000 [ 5988.496179] Lustre: server umount lustre-OST0000 complete [ 6004.195915] Lustre: 147770:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823293/real 1781823293] req@00000000fdb6f842 x1868376899107456/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1781823309 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6004.263723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.300828] Lustre: 147770:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6009.311239] Lustre: 147769:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823298/real 1781823298] req@000000007d6735d4 x1868376899108480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781823314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6009.395970] Lustre: 147769:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6014.949684] Lustre: 147768:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823303/real 1781823303] req@00000000b4da072e x1868376899109760/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781823319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6015.030576] Lustre: 147768:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6015.356029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.190705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.261889] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6020.256484] Lustre: 147769:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823309/real 1781823309] req@00000000ef2022e0 x1868376899110912/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781823325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6020.316669] Lustre: 147769:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6028.750473] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6029.432977] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6030.965942] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6030.969252] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6037.518616] Lustre: Failing over lustre-MDT0000 [ 6038.061355] Lustre: server umount lustre-MDT0000 complete [ 6038.567537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6038.579663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.599203] Lustre: Skipped 1 previous similar message [ 6040.059908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6040.119083] Lustre: Skipped 1 previous similar message [ 6045.156650] LustreError: 149566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.199470] LustreError: 149566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6046.271611] LustreError: 149560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.275616] LustreError: 149565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.329462] LustreError: 149565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6055.397157] LustreError: 149566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.461351] LustreError: 149566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6055.826465] Lustre: 147768:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823344/real 1781823344] req@00000000f36e5211 x1868376899124992/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781823360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6055.914734] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6061.029067] LustreError: 149565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.054572] LustreError: 149565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6067.727517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.265787] LustreError: 149565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.341541] LustreError: 149565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6082.603857] LustreError: 147766:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005da1e2f9 x1868376899132928/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6083.737767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.936868] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6087.233854] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6089.242117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6089.256385] Lustre: Skipped 1 previous similar message [ 6089.302220] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6089.377341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6098.078950] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6123.525563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6123.555475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.671394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6125.029506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6125.050548] Lustre: Skipped 1 previous similar message [ 6127.855150] Lustre: server umount lustre-OST0000 complete [ 6140.396840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.407661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6140.442935] Lustre: Skipped 2 previous similar messages [ 6140.487908] Lustre: Skipped 1 previous similar message [ 6145.359593] Lustre: server umount lustre-MDT0000 complete [ 6145.512463] LustreError: 149566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.550783] LustreError: 149566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6157.501541] LustreError: 149547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781823462 with bad export cookie 10159959193221238775 [ 6157.525793] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6157.560664] LustreError: 149547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6159.117224] Lustre: server umount lustre-MDT0001 complete [ 6191.098969] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6197.864864] Key type lgssc unregistered [ 6198.783119] LNet: 155246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6199.850676] LNet: Removed LNI 192.168.201.130@tcp [ 6202.356751] Key type .llcrypt unregistered [ 6202.384350] Key type ._llcrypt unregistered [ 6279.334908] Key type ._llcrypt registered [ 6279.347407] Key type .llcrypt registered [ 6279.986230] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6283.437716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6283.555526] alg: No test for adler32 (adler32-zlib) [ 6286.127859] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 6287.355929] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 6289.393356] Key type lgssc registered [ 6292.830065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6314.185991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6314.220926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.353090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.764923] LustreError: 156774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.871302] LustreError: 156774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6327.107746] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6331.893685] LustreError: 156775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.000905] LustreError: 156774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.070705] LustreError: 156775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.237728] LustreError: 156775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.276227] LustreError: 156775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6349.322796] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6357.477464] LustreError: 156774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.547419] LustreError: 156774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6359.784440] LustreError: 156803:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6359.832747] LustreError: 156803:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6359.855713] LustreError: 156803:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 43, retries 0, failed: rc = -5 [ 6360.679132] Lustre: server umount lustre-MDT0000 complete [ 6402.801084] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 6427.145815] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6517.206069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.926595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6518.111133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6518.413649] Lustre: lustre-MDT0000: new disk, initializing [ 6518.672285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6518.725403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6530.324140] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6556.379253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.880876] Lustre: 160156:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6557.085824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6557.105056] Lustre: Skipped 1 previous similar message [ 6557.389443] Lustre: lustre-MDT0001: new disk, initializing [ 6557.910090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6558.028980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6558.065122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6571.918781] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6594.868774] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6609.877775] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6628.071804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.940819] Lustre: lustre-OST0000: new disk, initializing [ 6628.951882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6628.963385] Lustre: 162137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6629.173951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6638.326702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6638.354569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6638.767094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6644.626402] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6666.290398] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6675.518059] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6679.843913] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:06:21 (1781823981) [ 7366.116305] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.119483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.180092] Lustre: Skipped 2 previous similar messages [ 7366.231047] Lustre: Skipped 2 previous similar messages [ 7371.233920] LustreError: 160165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.284762] LustreError: 160165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7373.112673] Lustre: server umount lustre-MDT0000 complete [ 7376.353418] LustreError: 160164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.410035] LustreError: 160164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7381.398202] LustreError: 160150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781824686 with bad export cookie 16931245343431504941 [ 7381.402824] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7381.415432] LustreError: 160150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7381.481773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7381.483421] LustreError: 162665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.517630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7386.611774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7388.064951] Lustre: server umount lustre-MDT0001 complete [ 7404.745568] Lustre: server umount lustre-OST0000 complete [ 7427.481331] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 7454.666117] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7545.201395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.693596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7545.771814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7546.028723] Lustre: lustre-MDT0000: new disk, initializing [ 7546.303068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.359659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7558.465145] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7585.177348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.358068] Lustre: 166905:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7585.544679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7585.558447] Lustre: Skipped 1 previous similar message [ 7585.743258] Lustre: lustre-MDT0001: new disk, initializing [ 7586.115822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.308392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7586.359810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7600.396894] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7627.008379] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7642.914325] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7664.829546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7666.004323] Lustre: lustre-OST0000: new disk, initializing [ 7666.027183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7666.043204] Lustre: 168883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7666.315280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.744196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7672.799579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7673.267678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7683.807708] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7707.755832] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7722.344323] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7723.043502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7738.196978] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7739.042432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7744.490392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.521516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.594860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.997125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7749.979812] Lustre: server umount lustre-OST0000 complete [ 7760.364341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7760.392565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.425979] Lustre: Skipped 1 previous similar message [ 7760.456276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.476065] Lustre: Skipped 3 previous similar messages [ 7766.193597] Lustre: server umount lustre-MDT0000 complete [ 7768.553717] LustreError: 169231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.628600] LustreError: 169231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7774.954801] LustreError: 167702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781825079 with bad export cookie 16931245343431663778 [ 7774.989367] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7774.998799] LustreError: 167702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7775.676392] Lustre: server umount lustre-MDT0001 complete [ 7796.347667] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:24:57 (1781825097) [ 7838.160372] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 7864.766677] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7955.746620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.373303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7956.547985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7956.766907] Lustre: lustre-MDT0000: new disk, initializing [ 7957.009920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.081428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7968.792916] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7991.231493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7991.631622] Lustre: 173684:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7991.809491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7991.829516] Lustre: Skipped 1 previous similar message [ 7992.023863] Lustre: lustre-MDT0001: new disk, initializing [ 7992.273671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7992.382474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7992.424318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8004.114629] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8026.404719] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8041.020811] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8058.782686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8059.808526] Lustre: lustre-OST0000: new disk, initializing [ 8059.817774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8059.845873] Lustre: 175671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8060.213561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.867334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8068.916334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8069.261283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8075.859542] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8098.916922] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8120.307983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8120.322300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.333631] LustreError: Skipped 1 previous similar message [ 8120.365572] Lustre: Skipped 1 previous similar message [ 8120.370911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8120.413501] Lustre: Skipped 3 previous similar messages [ 8123.104489] Lustre: server umount lustre-OST0000 complete [ 8135.649549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.662529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8135.688266] Lustre: Skipped 1 previous similar message [ 8138.943333] Lustre: server umount lustre-MDT0000 complete [ 8140.769043] LustreError: 173697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.847290] LustreError: 173697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8148.230097] LustreError: 173678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781825453 with bad export cookie 16931245343431664660 [ 8148.246690] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8148.252364] LustreError: 173678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8148.790453] Lustre: server umount lustre-MDT0001 complete [ 8169.008487] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:31:10 (1781825470) [ 8213.175689] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 8239.849709] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8329.457613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.940409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8329.992801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8330.166685] Lustre: lustre-MDT0000: new disk, initializing [ 8330.433314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.503407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8341.778727] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8367.636211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.994116] Lustre: 180037:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8368.249602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8368.271632] Lustre: Skipped 1 previous similar message [ 8368.618557] Lustre: lustre-MDT0001: new disk, initializing [ 8368.990157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8369.167540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8369.208563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8381.005949] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8403.563658] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8415.724729] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8433.012554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.783161] Lustre: lustre-OST0000: new disk, initializing [ 8433.790203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8433.798934] Lustre: 182027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8433.971305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.551592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8443.583493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8443.834759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8451.394878] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8472.073277] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8492.938868] Lustre: server umount lustre-OST0000 complete [ 8502.244858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8502.287633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8502.357053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8502.394842] Lustre: Skipped 2 previous similar messages [ 8510.367289] Lustre: 156086:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781825799/real 1781825799] req@0000000060e45789 x1868377455825152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781825815 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8512.816789] Lustre: server umount lustre-MDT0000 complete [ 8515.558564] LustreError: 181243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8515.626029] LustreError: 181243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8516.580180] Lustre: 156084:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781825805/real 1781825805] req@000000008621e707 x1868377455826944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781825821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8521.823427] Lustre: 156084:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781825810/real 1781825810] req@00000000bfe92c70 x1868377455828992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781825826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8522.412830] LustreError: 180029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781825827 with bad export cookie 16931245343431665745 [ 8522.421142] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.442910] LustreError: 180029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8523.404540] Lustre: server umount lustre-MDT0001 complete [ 8550.859193] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8558.490821] Key type lgssc unregistered [ 8559.308222] LNet: 183912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8560.362107] LNet: Removed LNI 192.168.201.130@tcp [ 8562.731726] Key type .llcrypt unregistered [ 8562.744717] Key type ._llcrypt unregistered [ 8612.527663] Key type ._llcrypt registered [ 8612.534406] Key type .llcrypt registered [ 8612.800993] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8615.849274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8616.002436] alg: No test for adler32 (adler32-zlib) [ 8617.981274] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 8618.627175] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8620.592458] Key type lgssc registered [ 8623.407773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8640.816818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8640.877895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.065633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.268268] LustreError: 185337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.329488] LustreError: 185337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8654.538340] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8657.419205] LustreError: 185338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.505083] LustreError: 185337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.626828] LustreError: 185338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8672.745195] LustreError: 185337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.845062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.484698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.821163] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8715.772910] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8728.136773] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8746.551972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.123936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8762.520735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8765.249921] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8786.122571] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8816.615600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8816.642472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.709570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8819.193572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8819.197065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8819.241160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.698983] Lustre: server umount lustre-OST0000 complete [ 8832.998119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8833.014868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.055397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8833.066976] Lustre: Skipped 1 previous similar message [ 8836.206606] Lustre: server umount lustre-MDT0000 complete [ 8839.651482] LustreError: 185333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.748406] LustreError: 185333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8845.544836] LustreError: 187856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781826150 with bad export cookie 2289572656442151960 [ 8845.548721] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8845.580401] LustreError: 187856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8846.437774] Lustre: server umount lustre-MDT0001 complete [ 8873.057434] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8878.513713] Key type lgssc unregistered [ 8879.153592] LNet: 188931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8880.264076] LNet: Removed LNI 192.168.201.130@tcp [ 8882.075579] Key type .llcrypt unregistered [ 8882.079915] Key type ._llcrypt unregistered [ 8942.389205] Key type ._llcrypt registered [ 8942.403922] Key type .llcrypt registered [ 8942.796244] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8945.964878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8946.086053] alg: No test for adler32 (adler32-zlib) [ 8948.228558] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 8949.060282] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8951.058215] Key type lgssc registered [ 8953.847075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8973.706038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8973.731405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.073690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.586168] LustreError: 190354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8985.644110] LustreError: 190354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8986.822204] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8990.691553] LustreError: 190353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.827840] LustreError: 190354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9000.938182] LustreError: 190353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.052187] LustreError: 190354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.911653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.183126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9019.424065] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9037.819543] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9052.902970] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9069.827319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9070.640521] LustreError: 192182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.099496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.275365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9087.407227] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9108.032258] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9121.422806] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 9122.275891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9122.302332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.353474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9124.354490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9124.361476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.523298] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 9126.553608] Lustre: Skipped 1 previous similar message [ 9131.596631] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 9131.628934] Lustre: Skipped 2 previous similar messages [ 9134.048075] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9134.521534] Lustre: server umount lustre-OST0000 complete [ 9155.171601] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff933190d99800.ost_server_uuid 50 [ 9159.816211] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 9183.206399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9183.226972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.277110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9187.997901] Lustre: server umount lustre-MDT0000 complete [ 9189.866497] LustreError: 190350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.913360] LustreError: 190350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9196.540878] LustreError: 190333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781826501 with bad export cookie 2100481071887136479 [ 9196.559080] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9196.563778] LustreError: 190333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9197.309510] Lustre: server umount lustre-MDT0001 complete [ 9219.248572] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:48:41 (1781826521) [ 9236.150700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.406723] LustreError: 194293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.487491] LustreError: 194293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9237.665426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.775738] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9268.260625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.419363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9280.733798] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9298.279501] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9313.569714] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9331.321100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9332.292896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9337.339926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9346.335290] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9365.810673] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9382.507524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9382.976341] Lustre: lustre-OST0001: new disk, initializing [ 9382.986953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9383.001321] Lustre: 197147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9383.245350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9391.932181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9391.970381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9392.224969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9399.151242] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9421.046464] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9429.477573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9429.514243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.584683] Lustre: Skipped 1 previous similar message [ 9429.615807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9429.626703] Lustre: Skipped 2 previous similar messages [ 9431.526221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.271166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9442.466789] Lustre: server umount lustre-OST0000 complete [ 9442.893687] LustreError: 196731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9442.933989] LustreError: 196731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9458.109484] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9458.661304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 9471.215960] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9471.794683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 9494.506021] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9494.528317] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.570198] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9494.572719] Lustre: Skipped 2 previous similar messages [ 9494.591416] Lustre: Skipped 8 previous similar messages [ 9499.674144] Lustre: server umount lustre-OST0001 complete [ 9509.860848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.863101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9509.944049] LustreError: Skipped 1 previous similar message [ 9516.871368] Lustre: server umount lustre-MDT0000 complete [ 9527.699881] LustreError: 194276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781826832 with bad export cookie 2100481071887137550 [ 9527.719336] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9527.765361] LustreError: 194276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9528.987682] Lustre: server umount lustre-MDT0001 complete [ 9587.907790] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:54:49 (1781826889) [ 9604.423098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.347092] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9605.487654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9606.063711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9616.714474] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9638.267056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9638.725248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9638.956220] Lustre: 200609:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9650.812722] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9669.212158] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9683.293528] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9702.549400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.252840] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9703.886152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.901620] Lustre: Skipped 1 previous similar message [ 9713.104866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9720.602378] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9739.514519] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9755.570670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9755.807980] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9772.492518] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9788.748161] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9799.668269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9799.703400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9799.716217] Lustre: Skipped 2 previous similar messages [ 9799.754381] Lustre: Skipped 7 previous similar messages [ 9801.896285] Lustre: server umount lustre-OST0000 complete [ 9804.829605] LustreError: 202682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9804.909952] LustreError: 202682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 9826.977844] Lustre: server umount lustre-OST0001 complete [ 9837.018787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.082200] Lustre: Skipped 1 previous similar message [ 9846.239459] Lustre: 189663:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781827135/real 1781827135] req@0000000048c19296 x1868380245325184/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781827151 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9846.473954] Lustre: server umount lustre-MDT0000 complete [ 9851.474096] Lustre: 189665:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781827140/real 1781827140] req@00000000aee13c45 x1868380245327232/t0(0) o400->lustre-OST0001-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781827156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9855.454241] LustreError: 201357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781827160 with bad export cookie 2100481071887138957 [ 9855.480463] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9855.480741] Lustre: 189666:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781827144/real 1781827144] req@00000000d4abe76c x1868380245329664/t0(0) o400->lustre-OST0001-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781827160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9855.480752] Lustre: 189666:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9855.490100] LustreError: 201357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9855.543953] LustreError: 189662:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a75b2c5a x1868380245331840/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9856.433440] Lustre: server umount lustre-MDT0001 complete [ 9915.321935] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:00:16 (1781827216) [ 9955.344734] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 9981.814243] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [10075.221089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.948582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10076.057421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10076.234839] Lustre: lustre-MDT0000: new disk, initializing [10076.411516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10076.422670] Lustre: Skipped 1 previous similar message [10076.487212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10087.178804] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10110.345617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.976206] Lustre: 208777:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10111.329603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10111.347913] Lustre: Skipped 1 previous similar message [10111.612484] Lustre: lustre-MDT0001: new disk, initializing [10112.059856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10112.126247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10122.134559] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10143.553931] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10159.693423] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10176.841549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10177.874850] Lustre: lustre-OST0000: new disk, initializing [10177.896588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10177.924329] Lustre: 210754:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10186.406902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10186.439475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10186.680670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10192.787988] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10217.167467] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10229.535814] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10230.073515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10241.672913] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10242.072755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10248.178115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.209888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10248.247978] Lustre: Skipped 2 previous similar messages [10248.281279] Lustre: Skipped 9 previous similar messages [10253.240957] Lustre: server umount lustre-OST0000 complete [10270.625691] Lustre: server umount lustre-MDT0000 complete [10273.765402] LustreError: 208788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.804347] LustreError: 208788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [10281.532370] LustreError: 208768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781827586 with bad export cookie 2100481071887140518 [10281.556864] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10281.588284] LustreError: 208768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10282.524586] Lustre: server umount lustre-MDT0001 complete [10304.425632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.893744] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10338.873449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.972727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10339.996402] Lustre: Skipped 3 previous similar messages [10351.100249] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10370.023474] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10384.784750] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10401.173686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.923957] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10438.604338] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10450.813386] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10451.489390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10463.860674] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10464.513323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10470.886396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10470.917764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.948250] Lustre: Skipped 2 previous similar messages [10484.193078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10484.639855] Lustre: server umount lustre-OST0000 complete [10499.890382] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10509.553837] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10549.232118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10562.687506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10562.704576] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10562.706174] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10564.830232] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10581.969526] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10585.448659] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10598.903438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10598.925619] Lustre: Skipped 11 previous similar messages [10603.623341] Lustre: server umount lustre-OST0000 complete [10619.473979] Lustre: server umount lustre-MDT0000 complete [10628.337429] LustreError: 212963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781827933 with bad export cookie 2100481071887141400 [10628.341630] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10628.368575] LustreError: 212963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10629.193864] Lustre: server umount lustre-MDT0001 complete [10647.683366] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:12:29 (1781827949) [10662.795250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.536826] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10695.986675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10708.694746] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10726.822117] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10740.687360] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10758.546897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10777.668949] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10797.761584] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10810.054241] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10810.929974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10823.657216] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10824.202845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10841.407851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10841.885261] Lustre: lustre-OST0001: new disk, initializing [10841.893704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10841.912788] Lustre: 222371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10851.628812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10851.651064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10851.912977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10858.394724] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10881.540199] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10892.829289] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [10893.277228] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10903.736544] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10904.182821] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10913.274775] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.280365] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10913.333514] Lustre: Skipped 6 previous similar messages [10913.352283] LustreError: Skipped 2 previous similar messages [10915.159590] Lustre: server umount lustre-OST0001 complete [10918.381925] LustreError: 220811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.475651] LustreError: 220811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [10931.875774] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [10932.531157] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [10943.737376] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10944.516664] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [10986.261679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10987.248299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10987.265287] Lustre: Skipped 6 previous similar messages [10992.653919] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [10992.703800] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [10992.709784] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10992.793977] Lustre: Skipped 2 previous similar messages [11003.652525] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11023.532890] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11028.367912] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11039.686971] Lustre: server umount lustre-OST0001 complete [11068.904369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11074.576946] Lustre: server umount lustre-OST0000 complete [11089.814990] Lustre: server umount lustre-MDT0000 complete [11099.071381] LustreError: 218961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781828403 with bad export cookie 2100481071887142366 [11099.091756] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11099.118958] LustreError: 218961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11105.990832] Lustre: server umount lustre-MDT0001 complete [11167.903772] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:21:08 (1781828468) [11184.233159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11185.109752] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11185.164799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11195.134748] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11217.281908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11217.473182] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11217.541336] Lustre: 227634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11229.738944] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11247.543494] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11262.328976] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11278.666766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.260963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11296.050128] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11315.036633] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11338.516562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11338.661906] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11354.946201] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11374.664135] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11379.390701] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11392.574056] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [11393.241339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11405.290259] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11406.148628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11422.724862] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9331874f8000.ost_server_uuid 50 [11427.143785] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9331874f8000.ost_server_uuid in IDLE state after 0 sec [11429.967178] Lustre: Permanently deactivating lustre-OST0001 [11429.991955] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [11440.362797] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [11440.602744] Lustre: Permanently reactivating lustre-OST0001 [11440.662531] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [11440.678939] Lustre: Skipped 2 previous similar messages [11448.077904] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.132261] Lustre: Skipped 6 previous similar messages [11448.153656] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [11448.182029] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [11448.242292] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [11448.379972] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [11452.438258] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11452.455981] Lustre: Skipped 17 previous similar messages [11456.902424] Lustre: server umount lustre-OST0001 complete [11472.783935] Lustre: server umount lustre-OST0000 complete [11489.112105] Lustre: server umount lustre-MDT0000 complete [11498.279304] LustreError: 227625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781828803 with bad export cookie 2100481071887143654 [11498.303279] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11498.335796] LustreError: 227625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11499.426899] Lustre: server umount lustre-MDT0001 complete [11553.594151] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:27:35 (1781828855) [11572.267547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.944431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11573.052845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11573.078413] Lustre: Skipped 2 previous similar messages [11585.110949] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11605.188660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.594232] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11605.698605] Lustre: 234376:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11606.458887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11606.483053] Lustre: Skipped 5 previous similar messages [11619.416571] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11639.096207] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11651.001680] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11666.793498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11667.363516] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11667.430335] Lustre: Permanently deactivating lustre-OST0000 [11667.482854] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11675.723724] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11675.739778] Lustre: Skipped 1 previous similar message [11681.887213] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11698.193314] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11713.062232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11713.473638] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11719.998165] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [11727.243771] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11744.442866] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11748.469296] Lustre: Permanently reactivating lustre-OST0000 [11748.537487] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11748.561545] Lustre: Skipped 2 previous similar messages [11756.749255] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [11756.753170] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 81 seconds. I think it's dead, and I am evicting it. exp 00000000456faf9b, cur 1781829061 deadline 1781829010 last 1781828980 [11756.771609] Lustre: Skipped 1 previous similar message [11756.773814] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [11756.845655] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [11756.863625] Lustre: Skipped 1 previous similar message [11756.981363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [11771.376770] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11772.810279] Lustre: server umount lustre-OST0001 complete [11776.490589] LustreError: 236232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.535174] LustreError: 236232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 61 previous similar messages [11787.760401] Lustre: server umount lustre-OST0000 complete [11795.938105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11795.950582] LustreError: Skipped 1 previous similar message [11802.372501] Lustre: server umount lustre-MDT0000 complete [11810.301503] LustreError: 234368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781829115 with bad export cookie 2100481071887145460 [11810.324744] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11810.342562] LustreError: 234368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11811.315374] Lustre: server umount lustre-MDT0001 complete [11838.665560] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11845.246349] Key type lgssc unregistered [11845.966478] LNet: 239296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11847.035320] LNet: Removed LNI 192.168.201.130@tcp [11849.236683] Key type .llcrypt unregistered [11849.241152] Key type ._llcrypt unregistered [11901.756723] Key type ._llcrypt registered [11901.764908] Key type .llcrypt registered [11902.087976] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11904.907749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11904.930886] alg: No test for adler32 (adler32-zlib) [11906.308704] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [11906.729728] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11908.551143] Key type lgssc registered [11911.611920] Lustre: Echo OBD driver; http://www.lustre.org/ [11933.771747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11933.822630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.328479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11946.469917] LustreError: 240771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11946.534357] LustreError: 240771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11947.405295] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11947.496144] LustreError: 240772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.635303] LustreError: 240771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.749853] LustreError: 240772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.851917] LustreError: 240771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.029447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11968.382677] Lustre: Found index 1 for lustre-MDT0001, updating log [11968.445449] Lustre: 240760:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11968.467702] Lustre: Permanently deactivating lustre-MDT0001 [11968.491791] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [11969.274035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11974.233399] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [11974.248803] LustreError: 240802:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [11974.312226] LustreError: 240802:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [11974.365878] LustreError: 240802:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 38, retries 1, failed: rc = -108 [11980.006981] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11997.582919] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12009.460384] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12022.998934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12023.825078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12033.075813] LustreError: 242619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.105814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [12033.116896] LustreError: 242619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12035.919176] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12039.188255] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [12049.406333] LustreError: 242619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12049.473588] LustreError: 242619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [12051.332950] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12066.046563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12066.409980] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12071.688100] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [12071.710198] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [12094.410649] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12128.183215] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12137.644376] Lustre: Permanently reactivating lustre-MDT0001 [12137.764482] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [12137.807308] Lustre: Skipped 2 previous similar messages [12139.867371] Lustre: lustre-MDT0001: Client 832356f1-582f-46ed-b5de-9e3eb6c8e89a (at 192.168.201.30@tcp) reconnecting [12159.016640] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12163.729121] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12181.244338] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12181.815566] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12194.308286] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12194.901938] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12201.259313] Lustre: Permanently deactivating lustre-MDT0001 [12201.290212] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [12201.300602] Lustre: Skipped 2 previous similar messages [12210.099842] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [12223.994113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12224.005275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12224.022690] Lustre: Skipped 2 previous similar messages [12224.055405] Lustre: Skipped 3 previous similar messages [12225.519844] Lustre: server umount lustre-MDT0000 complete [12229.090781] LustreError: 244415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12229.141621] LustreError: 244415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [12233.543574] LustreError: 240753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781829538 with bad export cookie 7947968962660505524 [12233.556420] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12233.574706] LustreError: 240753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12234.597416] Lustre: server umount lustre-MDT0001 complete [12244.263641] Lustre: server umount lustre-OST0000 complete [12269.757771] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:39:31 (1781829571) [12293.093718] Lustre: server umount lustre-OST0001 complete [12310.183205] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [12341.519346] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12437.936958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12438.850611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12438.909600] Lustre: Skipped 2 previous similar messages [12439.083682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12439.392410] Lustre: lustre-MDT0000: new disk, initializing [12439.724106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12439.812327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12452.462325] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12477.898255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.474167] Lustre: 249092:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12478.804416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12478.814492] Lustre: Skipped 1 previous similar message [12479.140805] Lustre: lustre-MDT0001: new disk, initializing [12479.478918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12479.591471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12479.654448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12491.185242] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12513.658881] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12528.904695] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12545.895282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12546.556486] Lustre: lustre-OST0000: new disk, initializing [12546.567427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12546.580358] Lustre: 251074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12546.775527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12555.932828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12555.952600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12556.256599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12562.872274] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12585.053902] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12589.967432] LustreError: 249100:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [12599.999621] LustreError: 249100:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [12600.107150] LustreError: 249101:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [12610.207446] LustreError: 249101:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [12611.148644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12611.581392] Lustre: lustre-OST0001: new disk, initializing [12611.602142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12611.625694] Lustre: 252196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12611.917279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12620.232505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12620.253910] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12620.528414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12630.732434] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12656.629265] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12662.389145] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12671.495691] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12671.497532] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12671.521462] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12671.522025] LustreError: Skipped 1 previous similar message [12671.614940] Lustre: Skipped 2 previous similar messages [12676.601795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12676.605479] Lustre: Skipped 1 previous similar message [12681.715735] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12681.766533] Lustre: Skipped 1 previous similar message [12682.208917] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12682.511963] Lustre: server umount lustre-OST0001 complete [12686.836520] LustreError: 251054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.890283] LustreError: 251054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12712.435261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.447343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12712.478606] Lustre: Skipped 1 previous similar message [12717.386581] Lustre: server umount lustre-OST0000 complete [12727.364271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12727.439423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12727.475963] Lustre: Skipped 2 previous similar messages [12734.132643] Lustre: server umount lustre-MDT0000 complete [12744.386659] LustreError: 249086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781830049 with bad export cookie 7947968962660508331 [12744.390597] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12744.426030] LustreError: 249086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12751.870541] Lustre: server umount lustre-MDT0001 complete [12781.758828] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12789.239588] Key type lgssc unregistered [12790.111240] LNet: 254265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12791.223726] LNet: Removed LNI 192.168.201.130@tcp [12793.530528] Key type .llcrypt unregistered [12793.552549] Key type ._llcrypt unregistered [12885.929769] Key type ._llcrypt registered [12885.936969] Key type .llcrypt registered [12886.320245] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12888.812413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12889.050136] alg: No test for adler32 (adler32-zlib) [12891.216768] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [12892.226711] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12894.263367] Key type lgssc registered [12897.569975] Lustre: Echo OBD driver; http://www.lustre.org/ [12918.358694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12918.420774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12920.308171] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12920.428526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12921.136562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12933.284150] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12954.439689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12954.682546] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12954.743723] Lustre: 256486:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12955.416962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12966.715916] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12983.871950] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12996.600127] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13011.906236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13012.566757] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13013.096456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13018.445099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [13028.900630] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13046.605147] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13131.236070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13131.241590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13131.283226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13131.302566] Lustre: Skipped 1 previous similar message [13133.013145] Lustre: server umount lustre-OST0000 complete [13143.334199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [13171.791136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13172.560907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13187.461931] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13197.050193] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [13197.093675] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [13197.097559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [13203.771579] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13256.694406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13256.717435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13256.741269] Lustre: Skipped 2 previous similar messages [13256.803219] Lustre: Skipped 1 previous similar message [13261.418298] Lustre: server umount lustre-OST0000 complete [13269.554414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13269.611204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13269.635665] Lustre: Skipped 1 previous similar message [13274.599041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13274.654652] Lustre: Skipped 1 previous similar message [13276.171528] Lustre: server umount lustre-MDT0000 complete [13281.761643] LustreError: 257360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13281.813241] LustreError: 257360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13284.484791] LustreError: 258356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781830589 with bad export cookie 5404117181515698748 [13284.487271] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13284.507679] LustreError: 258356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13285.367837] Lustre: server umount lustre-MDT0001 complete [13312.866149] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13319.372912] Key type lgssc unregistered [13319.982238] LNet: 261979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13321.074309] LNet: Removed LNI 192.168.201.130@tcp [13322.773379] Key type .llcrypt unregistered [13322.777651] Key type ._llcrypt unregistered [13376.510989] Key type ._llcrypt registered [13376.519930] Key type .llcrypt registered [13376.801903] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13379.087081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13379.110335] alg: No test for adler32 (adler32-zlib) [13381.323766] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [13381.993923] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13383.951283] Key type lgssc registered [13386.379429] Lustre: Echo OBD driver; http://www.lustre.org/ [13403.997303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13404.110962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13406.601946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13416.953904] LustreError: 263401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13417.031657] LustreError: 263401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13417.428857] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13422.055872] LustreError: 263400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.168878] LustreError: 263401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.283767] LustreError: 263400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13436.391658] LustreError: 263401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13437.253600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13437.825291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13447.933725] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13464.541404] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13479.971347] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13494.315539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13495.242943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13501.393040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [13507.807443] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13523.949437] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13570.033362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13570.047739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13570.079243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13570.116420] Lustre: Skipped 1 previous similar message [13574.706843] Lustre: server umount lustre-OST0000 complete [13582.430058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13582.438087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13582.451024] Lustre: Skipped 1 previous similar message [13582.456490] Lustre: Skipped 1 previous similar message [13587.431955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13587.460659] Lustre: Skipped 2 previous similar messages [13589.012274] Lustre: server umount lustre-MDT0000 complete [13589.557612] LustreError: 263400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13596.692330] LustreError: 263382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781830901 with bad export cookie 2163687252771537020 [13596.705826] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13596.748151] LustreError: 263382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13597.581982] Lustre: server umount lustre-MDT0001 complete [13622.526601] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13628.590664] Key type lgssc unregistered [13629.109750] LNet: 267759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13630.198630] LNet: Removed LNI 192.168.201.130@tcp [13631.982685] Key type .llcrypt unregistered [13631.993274] Key type ._llcrypt unregistered [13668.155845] Key type ._llcrypt registered [13668.165179] Key type .llcrypt registered [13668.399695] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13670.431991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13670.909299] alg: No test for adler32 (adler32-zlib) [13672.630274] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [13673.076861] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13674.792282] Key type lgssc registered [13677.180400] Lustre: Echo OBD driver; http://www.lustre.org/ [13694.234984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13694.295063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13696.758492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13706.210058] LustreError: 269001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13706.258520] LustreError: 269001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13706.351408] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13711.342069] LustreError: 269000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.453052] LustreError: 269001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.582712] LustreError: 269000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13724.809834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13725.606840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13734.231996] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13748.500252] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13760.396349] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13775.173881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13775.973899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13791.175478] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13791.363296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [13808.011994] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13831.663503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13831.665686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13831.682571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13831.708490] Lustre: Skipped 1 previous similar message [13831.724660] Lustre: Skipped 1 previous similar message [13834.187356] Lustre: server umount lustre-OST0000 complete [13841.894415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13841.895643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13841.923131] Lustre: Skipped 1 previous similar message [13841.935505] Lustre: Skipped 1 previous similar message [13847.014910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13847.030361] Lustre: Skipped 1 previous similar message [13848.042357] Lustre: server umount lustre-MDT0000 complete [13852.133989] LustreError: 269000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13852.186453] LustreError: 269000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13855.645724] LustreError: 269736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781831160 with bad export cookie 908780190345233759 [13855.657944] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13855.666359] LustreError: 269736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13856.506855] Lustre: server umount lustre-MDT0001 complete [13882.686435] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13889.611307] Key type lgssc unregistered [13890.306940] LNet: 272714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13891.366756] LNet: Removed LNI 192.168.201.130@tcp [13893.067584] Key type .llcrypt unregistered [13893.088438] Key type ._llcrypt unregistered [13939.023905] Key type ._llcrypt registered [13939.030663] Key type .llcrypt registered [13939.452493] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13941.715832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13941.748676] alg: No test for adler32 (adler32-zlib) [13943.351935] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [13943.828880] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13945.847891] Key type lgssc registered [13948.351968] Lustre: Echo OBD driver; http://www.lustre.org/ [13963.377368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13963.454982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13965.426687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13973.842929] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13976.033569] LustreError: 274137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13976.062981] LustreError: 274137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13981.164292] LustreError: 274138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13986.283620] LustreError: 274137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13990.071961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13990.401219] LustreError: 274138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13990.955841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13999.410576] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14012.384172] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14022.703132] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14034.044728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14034.643961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14035.752882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [14045.582104] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14059.068670] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14102.499508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14102.506540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14102.562968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14102.613026] LustreError: Skipped 1 previous similar message [14105.839564] Lustre: server umount lustre-OST0000 complete [14112.751585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14112.758543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14112.784684] Lustre: Skipped 2 previous similar messages [14112.822037] Lustre: Skipped 2 previous similar messages [14117.861222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14117.870643] Lustre: Skipped 1 previous similar message [14118.903322] Lustre: server umount lustre-MDT0000 complete [14122.978892] LustreError: 275025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14123.028665] LustreError: 275025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14124.970672] LustreError: 274875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781831429 with bad export cookie 13602051146706629061 [14124.989636] LustreError: 274875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14124.989722] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14125.698653] Lustre: server umount lustre-MDT0001 complete [14145.279574] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [14149.999963] Key type lgssc unregistered [14150.439992] LNet: 278474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14151.527936] LNet: Removed LNI 192.168.201.130@tcp [14153.092238] Key type .llcrypt unregistered [14153.095408] Key type ._llcrypt unregistered [14183.196909] Key type ._llcrypt registered [14183.208485] Key type .llcrypt registered [14183.406283] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14185.579448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14185.681483] alg: No test for adler32 (adler32-zlib) [14187.561869] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [14187.922434] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [14189.711956] Key type lgssc registered [14192.242534] Lustre: Echo OBD driver; http://www.lustre.org/ [14206.406565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14206.467630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14208.507509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14216.975188] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14218.732431] LustreError: 279721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14218.791955] LustreError: 279721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14223.844409] LustreError: 279723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14228.981131] LustreError: 279725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14231.040415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14231.772929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14238.905569] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14250.421957] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14260.789935] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14273.748168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14274.579950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14278.718355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [14285.941242] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14299.871589] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14319.589517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14319.619057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14319.639312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14322.672867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14322.681806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14322.705193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14325.122611] Lustre: server umount lustre-OST0000 complete [14332.899611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14332.907954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14332.940241] Lustre: Skipped 1 previous similar message [14332.979251] Lustre: Skipped 2 previous similar messages [14338.018905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14338.033429] Lustre: Skipped 1 previous similar message [14339.135909] Lustre: server umount lustre-MDT0000 complete [14343.137756] LustreError: 279729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14343.190893] LustreError: 279729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14344.929225] LustreError: 279688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781831649 with bad export cookie 1222788132801784349 [14344.934492] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14344.967895] LustreError: 279688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14345.788824] Lustre: server umount lustre-MDT0001 complete [14366.714621] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [14371.688683] Key type lgssc unregistered [14372.188205] LNet: 283454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14373.232567] LNet: Removed LNI 192.168.201.130@tcp [14374.589639] Key type .llcrypt unregistered [14374.597064] Key type ._llcrypt unregistered [14392.603701] Key type ._llcrypt registered [14392.607426] Key type .llcrypt registered [14392.814649] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [14421.404283] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [14443.844561] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14445.050948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14445.084341] alg: No test for adler32 (adler32-zlib) [14446.452792] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [14446.855853] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [14448.624086] Key type lgssc registered [14450.486162] Lustre: Echo OBD driver; http://www.lustre.org/ [14506.893254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14506.990275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14508.587701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14508.673417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14508.873138] Lustre: lustre-MDT0000: new disk, initializing [14509.104311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14509.147426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14516.688717] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14531.851794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14532.085399] Lustre: 287015:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14532.257762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14532.266041] Lustre: Skipped 1 previous similar message [14532.363681] Lustre: lustre-MDT0001: new disk, initializing [14532.624951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14532.679733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14532.704412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14541.135275] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14555.379686] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14564.139366] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14576.329727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14576.838970] Lustre: lustre-OST0000: new disk, initializing [14576.850146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14576.881728] Lustre: 288988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14577.107019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14584.465342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14584.485313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14584.640225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14587.615853] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14603.523373] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14613.011054] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14613.393577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14623.349449] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14623.973279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14630.895253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14630.901994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14630.950888] LustreError: Skipped 1 previous similar message [14630.952449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14630.997850] Lustre: Skipped 1 previous similar message [14634.215935] Lustre: server umount lustre-OST0000 complete [14645.224878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14645.260894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14645.333248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14645.353265] Lustre: Skipped 1 previous similar message [14647.989602] Lustre: server umount lustre-MDT0000 complete [14654.157453] LustreError: 287007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781831959 with bad export cookie 15106836764490112309 [14654.162284] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14654.179187] LustreError: 287007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14654.796705] Lustre: server umount lustre-MDT0001 complete [14669.785383] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:19:31 (1781831971) [14681.490169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14682.480922] LustreError: 291405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14682.525430] LustreError: 291405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14682.667178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14687.725328] LustreError: 291406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14690.782859] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14692.838199] LustreError: 291405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14697.965484] LustreError: 291406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14703.080224] LustreError: 291405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14705.320087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14706.054747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14712.846506] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14724.138756] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14732.110443] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14742.049449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14742.647843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14752.244966] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14763.110881] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14776.810126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14776.835850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14776.862285] Lustre: Skipped 1 previous similar message [14776.873387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14776.882289] Lustre: Skipped 2 previous similar messages [14778.864102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14778.878884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14778.895536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14778.901917] Lustre: Skipped 1 previous similar message [14781.524934] Lustre: server umount lustre-OST0000 complete [14788.066216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14788.102868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14788.133774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14793.853233] Lustre: server umount lustre-MDT0000 complete [14794.237276] LustreError: 291405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14794.266789] LustreError: 291405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14799.714441] LustreError: 293253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781832104 with bad export cookie 15106836764490113191 [14799.724573] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14799.742145] LustreError: 293253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14800.107907] Lustre: server umount lustre-MDT0001 complete [14821.340586] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [14826.075939] Key type lgssc unregistered [14826.552871] LNet: 294969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14827.647531] LNet: Removed LNI 192.168.201.130@tcp [14829.192795] Key type .llcrypt unregistered [14829.195787] Key type ._llcrypt unregistered [14870.899964] Key type ._llcrypt registered [14870.903922] Key type .llcrypt registered [14871.073343] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [14894.104695] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14895.279506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14895.390782] alg: No test for adler32 (adler32-zlib) [14896.913355] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [14897.231397] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [14898.962353] Key type lgssc registered [14900.900945] Lustre: Echo OBD driver; http://www.lustre.org/ [14965.168597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14965.223290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14966.680218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14966.760823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14966.925850] Lustre: lustre-MDT0000: new disk, initializing [14967.105526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14967.138873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14973.305646] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14987.154397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14987.364889] Lustre: 298801:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14987.476816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14987.481306] Lustre: Skipped 1 previous similar message [14987.604877] Lustre: lustre-MDT0001: new disk, initializing [14987.774347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14987.821465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14987.833789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14994.078430] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15006.589027] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15014.119066] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15023.935916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15024.370906] Lustre: lustre-OST2710: new disk, initializing [15024.377392] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [15024.391434] Lustre: 300771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15024.564503] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [15032.734558] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15033.973566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [15033.984927] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [15034.175347] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [15044.080200] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [15053.601345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15053.918846] Lustre: lustre-OST03e8: new disk, initializing [15053.927378] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [15053.943659] Lustre: 301837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15054.092689] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [15061.741845] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15063.696707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [15063.725699] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [15063.824447] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [15070.529099] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [15079.625482] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [15079.940236] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15086.006629] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [15086.379829] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15091.815758] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [15092.182879] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15097.451435] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [15097.803982] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15103.363826] Lustre: server umount lustre-MDT0000 complete [15105.014629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15105.029584] Lustre: Skipped 1 previous similar message [15107.857664] LustreError: 298793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781832412 with bad export cookie 18353958249338263264 [15107.858085] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15107.871148] LustreError: 298793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15108.386712] Lustre: server umount lustre-MDT0001 complete [15123.805550] Lustre: server umount lustre-OST2710 complete [15125.279560] Lustre: 296595:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781832414/real 1781832414] req@00000000ef46597c x1868386483117440/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781832430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15125.343181] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15125.367767] Lustre: Skipped 2 previous similar messages [15128.355505] Lustre: 296597:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781832417/real 1781832417] req@000000003e3e5536 x1868386483117696/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781832433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15129.145852] Lustre: server umount lustre-OST03e8 complete [15152.528430] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [15165.809635] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15219.038623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15219.578605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15219.649852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15219.779545] Lustre: lustre-MDT0000: new disk, initializing [15219.934156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15219.956911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15227.068941] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15242.091656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15242.303088] Lustre: 307094:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15242.399896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15242.408085] Lustre: Skipped 1 previous similar message [15242.609895] Lustre: lustre-MDT0001: new disk, initializing [15242.812368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15242.912482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15242.944706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15248.554806] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15261.937638] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15270.301919] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15280.135520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15280.534456] Lustre: lustre-OST0000: new disk, initializing [15280.538918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15280.550060] Lustre: 309066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15280.668558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15288.319500] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15288.424424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15288.439358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15288.614703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15300.954417] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15307.262705] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15307.613771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15313.764864] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15314.175444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15319.536278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15319.536161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15319.575451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15319.592507] LustreError: Skipped 1 previous similar message [15323.012488] Lustre: server umount lustre-OST0000 complete [15329.768435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15329.772104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15329.788680] Lustre: Skipped 1 previous similar message [15329.803295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15329.817614] Lustre: Skipped 2 previous similar messages [15334.769660] Lustre: server umount lustre-MDT0000 complete [15334.896235] LustreError: 307106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15334.919721] LustreError: 307106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [15340.005578] LustreError: 308169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15340.057828] LustreError: 308169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15340.682788] LustreError: 309065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781832645 with bad export cookie 18353958249338264944 [15340.694547] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15340.712850] LustreError: 309065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15341.260727] Lustre: server umount lustre-MDT0001 complete [15353.192653] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:30:56 (1781832656) [15355.388618] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [15358.481182] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:31:00 (1781832660) [15363.857940] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15402.447862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15402.883732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15402.932652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15403.259419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15409.468193] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15420.292054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15420.507833] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15420.566264] Lustre: 312929:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15426.832657] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15451.889357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15452.184953] Lustre: Denying initial registration attempt from nid 192.168.201.130@tcp, specified as failover [15452.198603] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [15452.209258] LustreError: 314384:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [15452.220679] LustreError: 314384:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [15452.227783] LustreError: 314384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [15452.342796] Lustre: server umount lustre-OST0000 complete [15452.349321] LustreError: 314384:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [15461.858134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15461.880860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15461.906264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15466.640413] Lustre: server umount lustre-MDT0000 complete [15468.001806] LustreError: 312933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15468.013953] LustreError: 312933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15471.526857] LustreError: 312919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781832776 with bad export cookie 18353958249338265826 [15471.532246] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15471.535840] LustreError: 312919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15471.939209] Lustre: server umount lustre-MDT0001 complete [15486.514415] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15490.067182] Key type lgssc unregistered [15490.459655] LNet: 315506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15491.495190] LNet: Removed LNI 192.168.201.130@tcp [15492.728366] Key type .llcrypt unregistered [15492.731386] Key type ._llcrypt unregistered [15508.084064] Key type ._llcrypt registered [15508.088882] Key type .llcrypt registered [15508.312166] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15510.094458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15510.133797] alg: No test for adler32 (adler32-zlib) [15511.392768] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [15511.732580] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15513.457894] Key type lgssc registered [15515.316819] Lustre: Echo OBD driver; http://www.lustre.org/ [15573.800972] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15586.716733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15586.761754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15588.344979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15588.396925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15588.635968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15595.289791] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15607.153261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15607.398560] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15607.453571] Lustre: 318289:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15607.762767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15612.927178] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15622.426799] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15629.578442] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15652.660410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15653.033581] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15653.367543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15662.737675] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15673.470193] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15678.434242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15678.446786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15678.468718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15678.953633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15678.974798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15682.531067] Lustre: server umount lustre-OST0000 complete [15689.186807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15689.191582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15689.219574] Lustre: Skipped 1 previous similar message [15689.259297] Lustre: Skipped 1 previous similar message [15694.359399] Lustre: server umount lustre-MDT0000 complete [15700.376908] LustreError: 318279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781833005 with bad export cookie 6073406764191281676 [15700.394795] LustreError: 318279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15700.398223] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15701.123975] Lustre: server umount lustre-MDT0001 complete [15716.436829] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15721.084722] Key type lgssc unregistered [15721.572743] LNet: 321866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15722.602961] LNet: Removed LNI 192.168.201.130@tcp [15723.895990] Key type .llcrypt unregistered [15723.898907] Key type ._llcrypt unregistered [15759.501240] Key type ._llcrypt registered [15759.504905] Key type .llcrypt registered [15759.722261] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15760.852833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15760.871937] alg: No test for adler32 (adler32-zlib) [15762.141669] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [15762.533390] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15764.283768] Key type lgssc registered [15765.913210] Lustre: Echo OBD driver; http://www.lustre.org/ [15777.033988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15777.055363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15778.948694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15785.009111] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15788.521883] LustreError: 323287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15788.550519] LustreError: 323287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15793.637599] LustreError: 323286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15795.586576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15796.130929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15801.261105] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15809.986862] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid