[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504769479 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.007735] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.014036] *** VALIDATE selinux *** [ 0.022459] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027630] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] *** VALIDATE tmpfs *** [ 0.031537] *** VALIDATE proc *** [ 0.032263] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035139] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037107] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039035] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.044302] debug: unmapping init [mem 0xffffffffad259000-0xffffffffad260fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047646] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054008] ... event mask: 000000070000000f [ 0.055270] rcu: Hierarchical SRCU implementation. [ 0.057218] smp: Bringing up secondary CPUs ... [ 0.058573] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.066427] smp: Brought up 1 node, 4 CPUs [ 0.068015] smpboot: Max logical packages: 1 [ 0.069010] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.129776] node 0 deferred pages initialised in 58ms [ 0.132463] devtmpfs: initialized [ 0.134282] x86/mm: Memory block size: 128MB [ 0.138306] gcov: version magic: 0x41383552 [ 0.141403] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146308] pinctrl core: initialized pinctrl subsystem [ 0.148170] [ 0.148730] ************************************************************* [ 0.151015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152011] ** ** [ 0.154012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.158010] ** This means that this kernel is built to expose internal ** [ 0.161012] ** IOMMU data structures, which may compromise security on ** [ 0.162011] ** your system. ** [ 0.165025] ** ** [ 0.167011] ** If you see this message and you are not debugging the ** [ 0.170013] ** kernel, report this immediately to your vendor! ** [ 0.172014] ** ** [ 0.175013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177012] ************************************************************* [ 0.179610] NET: Registered protocol family 16 [ 0.181417] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190058] cpuidle: using governor menu [ 0.191743] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194539] PCI: Using configuration type 1 for base access [ 0.197165] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207152] cryptd: max_cpu_qlen set to 1000 [ 0.209155] ACPI: Added _OSI(Module Device) [ 0.211012] ACPI: Added _OSI(Processor Device) [ 0.212022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219803] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225516] ACPI: Interpreter enabled [ 0.227061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228010] ACPI: Using IOAPIC for interrupt routing [ 0.229114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242057] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254332] acpiphp: Slot [2] registered [ 0.255109] acpiphp: Slot [5] registered [ 0.256116] acpiphp: Slot [6] registered [ 0.258110] acpiphp: Slot [7] registered [ 0.260159] acpiphp: Slot [8] registered [ 0.262124] acpiphp: Slot [9] registered [ 0.263121] acpiphp: Slot [10] registered [ 0.264078] acpiphp: Slot [3] registered [ 0.265104] acpiphp: Slot [4] registered [ 0.266085] acpiphp: Slot [11] registered [ 0.267082] acpiphp: Slot [12] registered [ 0.269136] acpiphp: Slot [13] registered [ 0.271410] acpiphp: Slot [14] registered [ 0.273097] acpiphp: Slot [15] registered [ 0.274094] acpiphp: Slot [16] registered [ 0.276104] acpiphp: Slot [17] registered [ 0.277187] acpiphp: Slot [18] registered [ 0.279148] acpiphp: Slot [19] registered [ 0.280097] acpiphp: Slot [20] registered [ 0.282100] acpiphp: Slot [21] registered [ 0.283091] acpiphp: Slot [22] registered [ 0.284107] acpiphp: Slot [23] registered [ 0.286096] acpiphp: Slot [24] registered [ 0.287107] acpiphp: Slot [25] registered [ 0.289114] acpiphp: Slot [26] registered [ 0.290097] acpiphp: Slot [27] registered [ 0.291079] acpiphp: Slot [28] registered [ 0.292063] acpiphp: Slot [29] registered [ 0.293080] acpiphp: Slot [30] registered [ 0.294090] acpiphp: Slot [31] registered [ 0.296079] PCI host bridge to bus 0000:00 [ 0.297014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.310845] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314045] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.324936] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.329013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334502] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.336675] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.339043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.342749] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.364694] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369747] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.474210] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.479012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.486014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.501016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.514000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.518015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.523013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540819] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.554849] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.564016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.575013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.602000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.610013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.619015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.653178] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.655358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.657231] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.659343] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.662238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.668040] iommu: Default domain type: Passthrough [ 0.669385] SCSI subsystem initialized [ 0.671134] ACPI: bus type USB registered [ 0.672129] usbcore: registered new interface driver usbfs [ 0.674064] usbcore: registered new interface driver hub [ 0.675079] usbcore: registered new device driver usb [ 0.677162] pps_core: LinuxPPS API ver. 1 registered [ 0.678010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.681059] PTP clock support registered [ 0.683135] EDAC MC: Ver: 3.0.0 [ 0.685047] PCI: Using ACPI for IRQ routing [ 0.686762] NetLabel: Initializing [ 0.688011] NetLabel: domain hash size = 128 [ 0.689010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691117] NetLabel: unlabeled traffic allowed by default [ 0.693282] vgaarb: loaded [ 0.695282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.697019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.705098] clocksource: Switched to clocksource kvm-clock [ 0.809690] VFS: Disk quotas dquot_6.6.0 [ 0.811838] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814672] *** VALIDATE ramfs *** [ 0.815972] *** VALIDATE hugetlbfs *** [ 0.817836] pnp: PnP ACPI init [ 0.820707] pnp: PnP ACPI: found 6 devices [ 0.838049] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.840545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.842361] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.844279] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.846161] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.848294] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.850486] NET: Registered protocol family 2 [ 0.852307] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.856254] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859690] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.865383] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.869362] TCP: Hash tables configured (established 65536 bind 65536) [ 0.872711] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.875745] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878375] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880989] NET: Registered protocol family 1 [ 0.883380] RPC: Registered named UNIX socket transport module. [ 0.885989] RPC: Registered udp transport module. [ 0.887577] RPC: Registered tcp transport module. [ 0.888672] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889943] NET: Registered protocol family 44 [ 0.890842] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.892094] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.894499] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.897308] PCI: CLS 0 bytes, default 64 [ 0.899840] Unpacking initramfs... [ 2.362435] debug: unmapping init [mem 0xffff94fcfcc54000-0xffff94fcfffbffff] [ 2.366507] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.368822] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.371937] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 4.359264] Initialise system trusted keyrings [ 4.365414] Key type blacklist registered [ 4.372729] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.396145] zbud: loaded [ 4.402915] *** VALIDATE nfs *** [ 4.409066] *** VALIDATE nfs4 *** [ 4.416649] pstore: using deflate compression [ 4.444438] Platform Keyring initialized [ 4.830452] NET: Registered protocol family 38 [ 4.834795] Key type asymmetric registered [ 4.836376] Asymmetric key parser 'x509' registered [ 4.839141] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.851626] io scheduler mq-deadline registered [ 4.855511] io scheduler kyber registered [ 4.858254] io scheduler bfq registered [ 4.867924] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.875753] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.878434] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.886919] ACPI: Power Button [PWRF] [ 4.898625] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.924561] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.957612] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.975806] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.003697] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.051223] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.120146] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.141252] Non-volatile memory driver v1.3 [ 5.142448] Linux agpgart interface v0.103 [ 5.239742] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 5.251232] vda: detected capacity change from 0 to 34816000 [ 5.288759] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.295765] vdb: detected capacity change from 0 to 1073741824 [ 5.336401] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.343798] vdc: detected capacity change from 0 to 2621440000 [ 5.369239] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.376039] vdd: detected capacity change from 0 to 2621440000 [ 5.413489] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.420418] vde: detected capacity change from 0 to 4294967296 [ 5.444192] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.448958] vdf: detected capacity change from 0 to 4294967296 [ 5.470980] libphy: Fixed MDIO Bus: probed [ 5.504578] usbcore: registered new interface driver usbserial_generic [ 5.509869] usbserial: USB Serial support registered for generic [ 5.513103] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.524403] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.530892] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.538400] mousedev: PS/2 mouse device common for all mice [ 5.542329] rtc_cmos 00:05: RTC can wake from S4 [ 5.545214] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.554282] rtc_cmos 00:05: registered as rtc0 [ 5.559088] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.568040] intel_pstate: CPU model not supported [ 5.569130] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.578409] hid: raw HID events driver (C) Jiri Kosina [ 5.582954] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.584708] usbcore: registered new interface driver usbhid [ 5.584715] usbhid: USB HID core driver [ 5.584884] drop_monitor: Initializing network drop monitor service [ 5.585039] Initializing XFRM netlink socket [ 5.585387] NET: Registered protocol family 10 [ 5.592534] Segment Routing with IPv6 [ 5.605257] NET: Registered protocol family 17 [ 5.609354] mpls_gso: MPLS GSO support [ 5.624119] RAS: Correctable Errors collector initialized. [ 5.626162] AVX version of gcm_enc/dec engaged. [ 5.627966] AES CTR mode by8 optimization enabled [ 5.891563] sched_clock: Marking stable (5891468521, 0)->(6760831465, -869362944) [ 5.897688] registered taskstats version 1 [ 5.901616] Loading compiled-in X.509 certificates [ 5.905342] zswap: loaded using pool lzo/zbud [ 5.962137] Key type big_key registered [ 5.980965] Key type encrypted registered [ 5.983414] ima: No TPM chip found, activating TPM-bypass! [ 5.986557] ima: Allocated hash algorithm: sha1 [ 5.988697] ima: No architecture policies found [ 5.990863] evm: Initialising EVM extended attributes: [ 5.993194] evm: security.selinux [ 5.994409] evm: security.ima [ 5.995857] evm: security.capability [ 5.997627] evm: HMAC attrs: 0x1 [ 6.001385] rtc_cmos 00:05: setting system clock to 2026-04-14 19:51:53 UTC (1776196313) [ 6.011315] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 6.015551] debug: unmapping init [mem 0xffffffffacf82000-0xffffffffad258fff] [ 6.024258] Write protecting the kernel read-only data: 28672k [ 6.031513] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 6.035606] debug: unmapping init [mem 0xffffffffabf14000-0xffffffffabffffff] [ 6.137515] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.158049] systemd[1]: Detected virtualization kvm. [ 6.168592] systemd[1]: Detected architecture x86-64. [ 6.173596] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.225960] systemd[1]: No hostname configured. [ 6.232114] systemd[1]: Set hostname to . [ 6.235584] random: systemd: uninitialized urandom read (16 bytes read) [ 6.239033] systemd[1]: Initializing machine ID from random generator. [ 6.382187] random: ln: uninitialized urandom read (6 bytes read) [ 6.577963] random: systemd: uninitialized urandom read (16 bytes read) [ 6.582163] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.595528] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.608794] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.948730] device-mapper: uevent: version 1.0.3 [ 7.953550] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.492825] virtio_net virtio0 ens2: renamed from eth0 [ 9.535864] random: fast init done [ 9.668258] scsi host0: ata_piix [ 9.769525] scsi host1: ata_piix [ 9.770906] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.781057] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.516337] random: crng init done [ 14.521672] random: 7 urandom warning(s) missed due to ratelimiting [ 15.030492] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.308153] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.776438] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.656710] SELinux: Disabled at runtime. [ 20.786941] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.804498] systemd[1]: Detected virtualization kvm. [ 20.808654] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.195891] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.208500] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.224849] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.232021] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.244826] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.267212] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.302956] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ 22.703407] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 23.614495] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.509842] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.560644] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.035822] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.150282] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *[0;1;3[ 30.824425] Key type dns_resolver registered 1m** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 31.622813] NFS: Registering the id_resolver key type [ 31.626142] Key type id_resolver registered [ 31.628375] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 47.943982] hrtimer: interrupt took 4971829 ns [ 87.914870] libcfs: loading out-of-tree module taints kernel. [ 87.954528] alg: No test for adler32 (adler32-zlib) [ 88.712323] Key type ._llcrypt registered [ 88.715440] Key type .llcrypt registered [ 88.837618] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 105.385196] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 106.602348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.255640] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 108.240089] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 108.245934] LNet: Accept secure, port 988 [ 110.063144] Key type lgssc registered [ 111.390264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.133826] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.065791] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.895466] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.281648] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 158.623305] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.961976] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 185.638874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.755870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.134575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 187.165062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 187.280785] Lustre: lustre-MDT0000: new disk, initializing [ 187.411502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.469813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.567392] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 205.234301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.335731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.418197] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 205.462286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 205.469314] Lustre: Skipped 1 previous similar message [ 205.526100] Lustre: lustre-MDT0001: new disk, initializing [ 205.603950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.625690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.636568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.706635] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 224.908094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.999062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.006519] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 225.156809] Lustre: lustre-OST0000: new disk, initializing [ 225.159649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 225.214457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.614937] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 241.170421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 241.264485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.281112] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 241.360934] Lustre: lustre-OST0001: new disk, initializing [ 241.364681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 241.421721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.589898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 242.600363] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 245.334047] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 255.607367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.263528] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 270.032936] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.216781] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 278.400860] Lustre: DEBUG MARKER: Client: 2.15.8.4 [ 280.733434] Lustre: DEBUG MARKER: MDS: 2.15.8.4 [ 283.034480] Lustre: DEBUG MARKER: OSS: 2.15.8.4 [ 284.505051] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 15:56:30 EDT 2026 [ 291.675809] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 293.030274] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 313.317310] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.329060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.356635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.346193] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.347829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.375221] Lustre: Skipped 2 previous similar messages [ 314.387693] Lustre: Skipped 3 previous similar messages [ 315.690851] Lustre: server umount lustre-MDT0000 complete [ 319.469086] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.492178] LustreError: Skipped 3 previous similar messages [ 320.101406] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196627 with bad export cookie 6736610391026556669 [ 320.105460] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.119226] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 324.575726] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.576630] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.577031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.601647] LustreError: Skipped 1 previous similar message [ 324.614972] Lustre: Skipped 1 previous similar message [ 325.600410] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.628853] LustreError: Skipped 1 previous similar message [ 326.587879] Lustre: server umount lustre-MDT0001 complete [ 330.474251] Lustre: server umount lustre-OST0000 complete [ 334.375652] Lustre: server umount lustre-OST0001 complete [ 340.629086] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 348.568882] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 358.111904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 363.684849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 369.430598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 374.483294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 380.542272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 380.626407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.871539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 380.913869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 381.001281] Lustre: lustre-MDT0000: new disk, initializing [ 381.091387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.113470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.032915] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 395.074698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 395.177831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.264765] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 395.308570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 395.313310] Lustre: Skipped 1 previous similar message [ 395.374541] Lustre: lustre-MDT0001: new disk, initializing [ 395.465533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 395.507487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 395.516925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 399.494615] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 408.485958] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 414.477784] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 420.673820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 420.740316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 420.745986] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 421.018756] Lustre: lustre-OST0000: new disk, initializing [ 421.021933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 421.136427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.989364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 423.000968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.816234] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 435.665446] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 439.832494] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 440.093411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 444.208580] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 444.367786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 448.993508] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.998377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.005404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 449.005413] Lustre: Skipped 3 previous similar messages [ 449.007828] LustreError: Skipped 1 previous similar message [ 449.050177] Lustre: Skipped 1 previous similar message [ 454.119875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.130483] Lustre: Skipped 2 previous similar messages [ 460.255181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 460.481241] Lustre: server umount lustre-OST0000 complete [ 464.357959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.365647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 464.381194] Lustre: Skipped 1 previous similar message [ 464.398974] Lustre: Skipped 3 previous similar messages [ 469.649799] Lustre: server umount lustre-MDT0000 complete [ 472.729129] LustreError: 14338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196780 with bad export cookie 6736610391026558125 [ 472.734826] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.742181] LustreError: 14338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 476.140157] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:59:42 (1776196782) [ 478.007515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 479.817399] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:59:45 (1776196785) [ 485.665505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.098317] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.161906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.462172] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 491.490555] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.035397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.724686] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 506.382317] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 511.740374] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 517.696244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.909954] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 528.151734] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 546.786052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.791689] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.815447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.827336] Lustre: Skipped 2 previous similar messages [ 551.668415] Lustre: server umount lustre-OST0000 complete [ 551.674875] Lustre: Skipped 1 previous similar message [ 562.147665] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.160262] LustreError: Skipped 1 previous similar message [ 565.428953] LustreError: 18873:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196872 with bad export cookie 6736610391026558979 [ 565.431463] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 565.441127] LustreError: 18873:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 579.920807] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 583.745826] Key type lgssc unregistered [ 584.314223] LNet: 22585:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.344231] LNet: Removed LNI 192.168.203.104@tcp [ 585.660455] Key type .llcrypt unregistered [ 585.666488] Key type ._llcrypt unregistered [ 611.075244] alg: No test for adler32 (adler32-zlib) [ 611.828469] Key type ._llcrypt registered [ 611.830719] Key type .llcrypt registered [ 611.916826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 613.035834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.615934] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 613.951078] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 613.961722] LNet: Accept secure, port 988 [ 615.615122] Key type lgssc registered [ 617.021359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 625.042272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.632661] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.707809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.428601] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 631.776478] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.901828] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.116221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.468244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 644.245678] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 652.315874] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 658.119936] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 664.408416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.803328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.365957] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 677.115236] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 677.540782] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 687.592173] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 687.593534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.599272] LustreError: Skipped 1 previous similar message [ 687.602076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 687.634054] Lustre: Skipped 1 previous similar message [ 692.711776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 692.720602] Lustre: Skipped 1 previous similar message [ 693.210776] Lustre: server umount lustre-OST0000 complete [ 697.826078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.830426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.847497] Lustre: Skipped 1 previous similar message [ 697.854112] Lustre: Skipped 1 previous similar message [ 702.758789] Lustre: server umount lustre-MDT0000 complete [ 702.947250] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.542885] LustreError: 24049:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197014 with bad export cookie 10434220567627802241 [ 706.550049] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.560522] LustreError: 24049:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 706.898502] Lustre: server umount lustre-MDT0001 complete [ 718.240282] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 720.618917] Key type lgssc unregistered [ 720.895410] LNet: 27724:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.908518] LNet: Removed LNI 192.168.203.104@tcp [ 721.789081] Key type .llcrypt unregistered [ 721.796905] Key type ._llcrypt unregistered [ 747.786565] alg: No test for adler32 (adler32-zlib) [ 748.548031] Key type ._llcrypt registered [ 748.553198] Key type .llcrypt registered [ 748.678918] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 749.738453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 750.776746] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 750.982095] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 750.985661] LNet: Accept secure, port 988 [ 752.665441] Key type lgssc registered [ 753.663569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 761.537462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.147198] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.197648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 767.965233] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 768.483519] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.605440] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 775.727693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.026285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 779.549748] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 785.647841] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 790.806313] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 796.648174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 796.912716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.879835] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 809.390500] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 813.318859] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 820.739596] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 821.687766] Lustre: server umount lustre-MDT0000 complete [ 822.241653] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 822.258605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.287190] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.783775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.793688] Lustre: Skipped 1 previous similar message [ 825.119282] LustreError: 31869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197132 with bad export cookie 73982001123596019 [ 825.132037] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 825.135569] LustreError: 31869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 825.845583] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 828.896569] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.901118] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.928198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 828.951777] LustreError: Skipped 4 previous similar messages [ 830.946122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 830.948803] Lustre: Skipped 1 previous similar message [ 831.747244] Lustre: server umount lustre-MDT0001 complete [ 854.371585] Lustre: server umount lustre-OST0000 complete [ 872.259036] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 874.653190] Key type lgssc unregistered [ 874.914612] LNet: 33201:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.919746] LNet: Removed LNI 192.168.203.104@tcp [ 875.664484] Key type .llcrypt unregistered [ 875.667708] Key type ._llcrypt unregistered [ 897.222481] alg: No test for adler32 (adler32-zlib) [ 897.983086] Key type ._llcrypt registered [ 897.987336] Key type .llcrypt registered [ 898.109955] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 899.153469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.445895] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 899.702869] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 899.706679] LNet: Accept secure, port 988 [ 901.367124] Key type lgssc registered [ 902.478372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 909.018603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.372302] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.423043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.455561] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 915.431202] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.550969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.882923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.177480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 926.978741] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 934.683695] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 940.907522] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 949.364828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.755665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.751648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 959.997612] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 962.806502] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 970.214044] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.214436] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 970.219499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.260816] Lustre: Skipped 2 previous similar messages [ 975.322705] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 975.336601] Lustre: Skipped 2 previous similar messages [ 975.688781] Lustre: server umount lustre-OST0000 complete [ 993.763434] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 993.768552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.777534] Lustre: Skipped 1 previous similar message [ 993.781022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.813115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.818978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.886851] Lustre: server umount lustre-MDT0000 complete [ 1000.930686] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.943437] LustreError: Skipped 1 previous similar message [ 1003.970718] LustreError: 34649:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197311 with bad export cookie 7363848589650378989 [ 1003.976177] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.984366] LustreError: 34649:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1018.335202] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1018.676130] Lustre: server umount lustre-MDT0001 complete [ 1029.025197] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1031.413682] Key type lgssc unregistered [ 1031.667857] LNet: 38502:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.679771] LNet: Removed LNI 192.168.203.104@tcp [ 1032.433229] Key type .llcrypt unregistered [ 1032.436969] Key type ._llcrypt unregistered [ 1052.346870] alg: No test for adler32 (adler32-zlib) [ 1053.105086] Key type ._llcrypt registered [ 1053.107292] Key type .llcrypt registered [ 1053.246510] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1054.161060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.756743] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 1055.119162] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1055.122588] LNet: Accept secure, port 988 [ 1056.823135] Key type lgssc registered [ 1057.811756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.198547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.660384] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.744570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.962680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1072.100733] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.236135] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.692700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.169291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.224602] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1096.724677] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1103.774704] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1112.156719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.510380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.597211] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1119.195337] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1128.494885] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1137.497309] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1140.873056] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1145.865449] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1145.882949] LNetError: 39265:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1145.903617] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1153.346827] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1416h10m58s [ 1155.320072] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1157.099063] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1157.107473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.132969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.186087] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1161.708151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.720270] Lustre: Skipped 1 previous similar message [ 1161.737296] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1161.749141] LNetError: 39263:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1161.756604] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1163.341554] Lustre: server umount lustre-MDT0000 complete [ 1166.283463] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.824125] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1166.850152] LNetError: 39262:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1166.854615] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1167.481320] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1171.401645] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.426416] LustreError: Skipped 3 previous similar messages [ 1171.990763] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1172.002567] LNetError: 39264:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1172.024549] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1173.984642] Lustre: 39277:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197474/real 1776197474] req@00000000c8ae65ef x1862477925077888/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776197481 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1174.031754] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.445965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.662077] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.689345] LustreError: Skipped 11 previous similar messages [ 1180.140686] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xbbdca8e4f4051e73 to 0xbbdca8e4f40522d3 [ 1180.160527] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 1180.412771] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1180.418358] LNetError: 39265:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1180.423094] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1180.436456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.495314] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1185.564563] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1185.776706] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1185.807866] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1185.823276] Lustre: Skipped 2 previous similar messages [ 1189.169818] Lustre: DEBUG MARKER: Wait for df (28937) ... [ 1190.744709] Lustre: DEBUG MARKER: done [ 1196.014025] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.016775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.027960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1196.044408] Lustre: Skipped 1 previous similar message [ 1196.056495] Lustre: Skipped 4 previous similar messages [ 1201.123099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1201.132187] Lustre: Skipped 1 previous similar message [ 1206.240368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.245095] Lustre: Skipped 1 previous similar message [ 1209.311112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1209.472434] Lustre: server umount lustre-OST0000 complete [ 1216.481561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.485146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1216.496299] Lustre: Skipped 1 previous similar message [ 1216.499877] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1216.507574] Lustre: Skipped 1 previous similar message [ 1216.525378] LNetError: 39263:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1216.537237] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1219.919122] Lustre: server umount lustre-MDT0000 complete [ 1221.600673] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.616970] LustreError: Skipped 4 previous similar messages [ 1223.410956] LustreError: 39945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197530 with bad export cookie 13536880281268855507 [ 1223.413552] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.416797] LustreError: 39945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1223.729844] Lustre: server umount lustre-MDT0001 complete [ 1235.604962] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1238.033427] Key type lgssc unregistered [ 1238.262608] LNet: 45071:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.274559] LNet: Removed LNI 192.168.203.104@tcp [ 1239.144948] Key type .llcrypt unregistered [ 1239.149098] Key type ._llcrypt unregistered [ 1273.707373] alg: No test for adler32 (adler32-zlib) [ 1274.466744] Key type ._llcrypt registered [ 1274.469886] Key type .llcrypt registered [ 1274.569268] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1288.808685] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1290.120086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.542444] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 1290.813864] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1290.817252] LNet: Accept secure, port 988 [ 1292.479216] Key type lgssc registered [ 1293.593936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1302.393494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1309.227604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1315.657818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1322.326820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1328.805873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1328.871665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.044793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1330.081475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1330.162867] Lustre: lustre-MDT0000: new disk, initializing [ 1330.225272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.249277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1333.971925] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1344.031551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1344.120458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.246447] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1344.289464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1344.293535] Lustre: Skipped 1 previous similar message [ 1344.370487] Lustre: lustre-MDT0001: new disk, initializing [ 1344.439803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.475471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1344.487597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1350.738022] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1361.993900] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1368.175305] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1375.426675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1375.556547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1375.570150] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1375.746662] Lustre: lustre-OST0000: new disk, initializing [ 1375.753281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1375.840738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.763660] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1386.073314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1386.079234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1392.786312] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1398.634646] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1398.844564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1403.549052] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1403.750655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1406.948457] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1406.950220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.954988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1406.961784] LustreError: Skipped 1 previous similar message [ 1406.992895] Lustre: Skipped 1 previous similar message [ 1412.068948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.079548] Lustre: Skipped 2 previous similar messages [ 1417.199432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.209850] Lustre: Skipped 1 previous similar message [ 1420.255402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1420.481796] Lustre: server umount lustre-OST0000 complete [ 1426.400564] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1426.405282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.412869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.442767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.682281] Lustre: server umount lustre-MDT0000 complete [ 1432.545576] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.554792] LustreError: Skipped 1 previous similar message [ 1434.716702] LustreError: 52869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197742 with bad export cookie 10582549011577894354 [ 1434.723681] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.726476] LustreError: 52869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1435.038791] Lustre: server umount lustre-MDT0001 complete [ 1443.014491] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 16:15:49 (1776197749) [ 1450.104977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.913143] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.995987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.511178] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1456.098168] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.218262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.902529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1465.466343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1470.438939] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1478.539359] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.056096] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1489.812565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1490.122050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.182648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1501.585774] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1507.392901] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1416h16m53s [ 1508.659368] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1516.040550] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1518.336838] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1518.403306] LustreError: 55689:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1519.607509] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1519.616802] LNetError: 47393:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1519.628495] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1538.600681] Lustre: lustre-MDT0000: Client 080d53a8-a209-40da-aa63-7bb5298ad9e1 (at 192.168.203.4@tcp) reconnecting [ 1558.432361] LustreError: 55689:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1560.472944] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1562.716438] Lustre: DEBUG MARKER: done [ 1568.224412] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1568.229516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.237995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1568.249234] Lustre: Skipped 1 previous similar message [ 1568.263648] Lustre: Skipped 3 previous similar messages [ 1570.164906] Lustre: server umount lustre-OST0000 complete [ 1577.953516] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1577.959090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.975465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.468824] LNetError: 120-3: Refusing connection from 192.168.203.104 for 127.0.0.2@tcp: No matching NI [ 1578.478203] LNetError: 47394:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1578.485825] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1579.828857] Lustre: server umount lustre-MDT0000 complete [ 1583.538277] LustreError: 53722:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197891 with bad export cookie 10582549011577895208 [ 1583.542922] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.549176] LustreError: 53722:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1595.122931] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1597.820915] Key type lgssc unregistered [ 1598.110289] LNet: 58155:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.116992] LNet: Removed LNI 192.168.203.104@tcp [ 1598.900241] Key type .llcrypt unregistered [ 1598.902836] Key type ._llcrypt unregistered [ 1625.567128] alg: No test for adler32 (adler32-zlib) [ 1626.327483] Key type ._llcrypt registered [ 1626.331532] Key type .llcrypt registered [ 1626.447777] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1638.200436] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1639.319420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.603962] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 1639.771726] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1639.775734] LNet: Accept secure, port 988 [ 1641.447171] Key type lgssc registered [ 1642.405649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.395327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1656.511370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1662.609903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1669.069746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1675.842366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1675.939033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.173962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1677.205547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1677.299587] Lustre: lustre-MDT0000: new disk, initializing [ 1677.396122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.412409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1681.674781] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1692.722623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1692.801695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.907812] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1692.965252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1692.973756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1692.984299] Lustre: Skipped 1 previous similar message [ 1693.073642] Lustre: lustre-MDT0001: new disk, initializing [ 1693.137640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1693.168731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1693.177816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1697.732435] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1708.171818] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1713.796431] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1719.854554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1719.927932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.938723] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1720.166890] Lustre: lustre-OST0000: new disk, initializing [ 1720.171876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1720.292232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.682407] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1726.526150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1726.538841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1734.102605] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1738.804481] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1739.092108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1743.667273] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1743.896525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1747.429886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.433399] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.444552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.225280] Lustre: server umount lustre-OST0000 complete [ 1757.664971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.678189] Lustre: Skipped 1 previous similar message [ 1757.682795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.688512] Lustre: Skipped 2 previous similar messages [ 1762.713330] Lustre: server umount lustre-MDT0000 complete [ 1762.795126] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.809311] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.838583] LustreError: Skipped 2 previous similar messages [ 1767.407863] LustreError: 62246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198074 with bad export cookie 3031160519903949091 [ 1767.426235] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1767.432253] LustreError: 62246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1767.969745] Lustre: server umount lustre-MDT0001 complete [ 1776.300610] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 16:21:22 (1776198082) [ 1777.858724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1779.673213] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 16:21:25 (1776198085) [ 1791.609473] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1795.722310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.141375] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.207341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.160704] Lustre: Failing over lustre-MDT0000 [ 1797.168736] LustreError: 67268:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1797.180599] LustreError: 67268:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1797.189320] LustreError: 67268:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1802.719433] Lustre: 67319:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198104/real 1776198104] req@00000000ffc3eb27 x1862478538495552/t0(0) o251->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198110 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1802.902760] Lustre: server umount lustre-MDT0000 complete [ 1808.626331] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:21:54 (1776198114) [ 1814.909602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.348841] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.412113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.923493] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1820.652312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.473597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.901735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.808911] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1841.380912] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1847.232992] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1853.491461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.778896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.680505] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1867.441690] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1873.610587] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1881.574576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.578614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.597533] Lustre: Skipped 3 previous similar messages [ 1881.605412] Lustre: Skipped 2 previous similar messages [ 1886.351909] Lustre: server umount lustre-MDT0000 complete [ 1886.696646] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.733412] LustreError: Skipped 3 previous similar messages [ 1890.215381] LustreError: 67867:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198197 with bad export cookie 3031160519903950211 [ 1890.223260] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.236221] LustreError: 67867:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1894.682419] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1898.975144] Lustre: 60474:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198199/real 1776198199] req@00000000fe7d25be x1862478538514688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198206 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1898.998499] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.065180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1903.074827] Lustre: 60471:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198203/real 1776198203] req@00000000f7621d63 x1862478538514816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198210 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1909.858183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.608591] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x2a10d86ede103583 to 0x2a10d86ede103d24 [ 1922.631078] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 1923.071453] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.189579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.006664] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1929.703519] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1936.093959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.441383] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1940.827249] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 1941.494564] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1948.172555] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1953.713572] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1960.329510] Lustre: server umount lustre-MDT0000 complete [ 1960.333528] Lustre: Skipped 1 previous similar message [ 1961.955981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.965370] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.989672] Lustre: Skipped 1 previous similar message [ 1962.004975] LustreError: Skipped 4 previous similar messages [ 1963.528643] LustreError: 70838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198271 with bad export cookie 3031160519903952164 [ 1963.531937] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.557522] LustreError: 70838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.077416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1976.154239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1985.082634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.771725] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x2a10d86ede103d24 to 0x2a10d86ede10432f [ 1996.785374] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 1997.186441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.205892] Lustre: Skipped 1 previous similar message [ 1997.280816] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2001.865251] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2011.568275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.268668] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 2016.343299] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2019.056560] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2030.193957] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2037.498566] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2046.946654] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.947973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.962979] Lustre: Skipped 4 previous similar messages [ 2046.974498] Lustre: Skipped 2 previous similar messages [ 2051.597445] Lustre: server umount lustre-MDT0000 complete [ 2051.604191] Lustre: Skipped 1 previous similar message [ 2052.069817] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.089182] LustreError: Skipped 10 previous similar messages [ 2055.913146] LustreError: 70838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198363 with bad export cookie 3031160519903953711 [ 2055.924607] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.931462] LustreError: 70838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2062.544348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2064.223202] Lustre: 60473:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198364/real 1776198364] req@000000000af42b2a x1862478538542976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198371 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2064.253757] Lustre: 60473:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2064.263104] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.158381] Lustre: DEBUG MARKER: files compared the same [ 2084.585643] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2087.010447] Key type lgssc unregistered [ 2087.268503] LNet: 77929:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.285192] LNet: Removed LNI 192.168.203.104@tcp [ 2087.915798] Key type .llcrypt unregistered [ 2087.917984] Key type ._llcrypt unregistered [ 2108.699634] alg: No test for adler32 (adler32-zlib) [ 2109.460116] Key type ._llcrypt registered [ 2109.465167] Key type .llcrypt registered [ 2109.593674] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2110.510231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.048104] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 2111.259513] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2111.264955] LNet: Accept secure, port 988 [ 2112.927111] Key type lgssc registered [ 2113.871604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2120.960294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.380520] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.456951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.899226] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all 8 [ 2127.840950] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.963493] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.671487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.867671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.106506] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all 8 [ 2143.266813] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2148.601324] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2154.575845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2154.815491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.561310] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all 8 [ 2164.016047] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 2164.868412] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2173.221450] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2183.650525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.673313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.676269] Lustre: Skipped 1 previous similar message [ 2187.801358] Lustre: server umount lustre-OST0000 complete [ 2193.888057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.901562] Lustre: Skipped 1 previous similar message [ 2193.915745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.930653] Lustre: Skipped 1 previous similar message [ 2198.160920] Lustre: server umount lustre-MDT0000 complete [ 2199.022993] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.053870] LustreError: Skipped 1 previous similar message [ 2203.042521] LustreError: 79373:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198510 with bad export cookie 10906146821953720445 [ 2203.046444] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.049974] LustreError: 79373:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2203.335941] Lustre: server umount lustre-MDT0001 complete [ 2218.775662] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2223.114540] Key type lgssc unregistered [ 2223.601753] LNet: 83066:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.616789] LNet: Removed LNI 192.168.203.104@tcp [ 2224.960831] Key type .llcrypt unregistered [ 2224.971991] Key type ._llcrypt unregistered [ 2248.959924] alg: No test for adler32 (adler32-zlib) [ 2249.718233] Key type ._llcrypt registered [ 2249.721754] Key type .llcrypt registered [ 2249.830415] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2250.678939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.018760] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 2251.284203] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2251.289912] LNet: Accept secure, port 988 [ 2253.008321] Key type lgssc registered [ 2253.953586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2261.005302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2278.559364] LustreError: 84502:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 1: rc = -5 [ 2309.279224] LustreError: 84502:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2339.106753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.027705] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2350.551494] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2358.777686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.881994] Lustre: *** cfs_fail_loc=706, val=0*** [ 2361.935184] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.014429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.070988] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 2366.109181] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2367.465324] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.578583] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.705651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.080383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2374.128525] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 2378.414918] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2385.979330] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2392.421985] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2394.593761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.595115] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.615239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.724494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.730474] Lustre: Skipped 1 previous similar message [ 2400.419785] Lustre: server umount lustre-OST0000 complete [ 2404.847426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.850912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.869434] Lustre: Skipped 2 previous similar messages [ 2404.894727] Lustre: Skipped 2 previous similar messages [ 2409.953403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.955968] Lustre: Skipped 1 previous similar message [ 2411.007764] Lustre: server umount lustre-MDT0000 complete [ 2414.860351] LustreError: 84509:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198722 with bad export cookie 12098817568464586891 [ 2414.868868] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.874419] LustreError: 84509:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2415.379627] Lustre: server umount lustre-MDT0001 complete [ 2426.263485] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2428.866465] Key type lgssc unregistered [ 2429.237303] LNet: 88243:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.251785] LNet: Removed LNI 192.168.203.104@tcp [ 2430.030108] Key type .llcrypt unregistered [ 2430.034069] Key type ._llcrypt unregistered [ 2452.143698] alg: No test for adler32 (adler32-zlib) [ 2452.903281] Key type ._llcrypt registered [ 2452.907151] Key type .llcrypt registered [ 2453.022630] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2453.914987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2454.495685] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 2454.681877] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2454.687424] LNet: Accept secure, port 988 [ 2456.330140] Key type lgssc registered [ 2457.181478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.870959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.664561] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2477.053995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.361184] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.440301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.471063] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2482.658294] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.670466] LustreError: Skipped 2 previous similar messages [ 2487.780994] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.802613] LustreError: Skipped 1 previous similar message [ 2489.382570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.709627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.771962] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.794759] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2492.797886] LustreError: Skipped 5 previous similar messages [ 2493.766517] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2498.018632] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.032472] LustreError: Skipped 2 previous similar messages [ 2501.593382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2501.602162] Lustre: Skipped 1 previous similar message [ 2501.743245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.799869] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2505.339831] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2516.962332] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.976995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.996750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2517.480548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.598105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.609460] Lustre: Skipped 2 previous similar messages [ 2522.928390] Lustre: server umount lustre-OST0000 complete [ 2527.719514] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2527.720379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.726207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.726215] Lustre: Skipped 1 previous similar message [ 2527.765048] Lustre: Skipped 1 previous similar message [ 2532.456881] Lustre: server umount lustre-MDT0000 complete [ 2532.832785] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.840214] LustreError: Skipped 2 previous similar messages [ 2535.885401] Lustre: server umount lustre-MDT0001 complete [ 2545.119505] Lustre: 93111:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198846/real 1776198846] req@00000000a5fc7c27 x1862479393076032/t0(0) o251->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198852 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2545.156430] Lustre: server umount MGS complete [ 2554.304310] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:34:20 (1776198860) [ 2577.903663] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2584.146971] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2590.876689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2596.252259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.607190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2607.202444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2614.099291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2614.176269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.297414] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2629.852406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.029827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.225608] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2630.254642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2630.361673] Lustre: lustre-MDT0001: new disk, initializing [ 2630.455444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2630.487162] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2630.499580] LustreError: 97811:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2636.238294] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2648.092691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2648.176887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.189267] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2648.613040] Lustre: lustre-OST0000: new disk, initializing [ 2648.620579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2648.699421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.359309] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2659.871881] Lustre: 99105:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000b30c0811 x1862479393080896/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776198979 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2663.860474] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2670.121742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2670.162470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2670.245324] Lustre: lustre-MDT0000: new disk, initializing [ 2670.381972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.408719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2671.457152] Lustre: 99664:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000df8ce706 x1862479393084352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776198990 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2673.508309] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2674.586049] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2680.305368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2680.325311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2705.377989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.384384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2705.405499] Lustre: Skipped 1 previous similar message [ 2705.424970] Lustre: Skipped 2 previous similar messages [ 2715.618793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2715.633836] Lustre: Skipped 3 previous similar messages [ 2718.175289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2718.331241] Lustre: server umount lustre-OST0000 complete [ 2725.857402] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2725.857612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.890153] Lustre: Skipped 1 previous similar message [ 2728.581046] Lustre: server umount lustre-MDT0000 complete [ 2730.976148] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.998693] LustreError: Skipped 1 previous similar message [ 2748.895151] Lustre: 100858:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199050/real 1776199050] req@0000000070477a23 x1862479393096064/t0(0) o251->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776199056 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2748.971617] Lustre: server umount MGS complete [ 2748.973505] Lustre: Skipped 1 previous similar message [ 2761.789192] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:37:47 (1776199067) [ 2783.608657] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2786.633408] Key type lgssc unregistered [ 2786.974832] LNet: 102269:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.995460] LNet: Removed LNI 192.168.203.104@tcp [ 2787.799733] Key type .llcrypt unregistered [ 2787.810391] Key type ._llcrypt unregistered [ 2802.579163] alg: No test for adler32 (adler32-zlib) [ 2803.335926] Key type ._llcrypt registered [ 2803.338339] Key type .llcrypt registered [ 2803.452802] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2804.689863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.066516] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 2805.310475] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2805.316758] LNet: Accept secure, port 988 [ 2806.991957] Key type lgssc registered [ 2808.256348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2815.464287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.892254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.944505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.845835] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2822.115544] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.233516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.813105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.253147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2832.100794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2839.221861] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2844.604817] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2851.257382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.641090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.689573] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2863.172563] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2866.996696] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2877.412844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.413279] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.434570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.464090] Lustre: Skipped 1 previous similar message [ 2880.167353] Lustre: server umount lustre-OST0000 complete [ 2884.576112] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.582713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.605221] Lustre: Skipped 1 previous similar message [ 2884.612992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.652562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.653560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.672113] Lustre: Skipped 1 previous similar message [ 2889.648079] Lustre: server umount lustre-MDT0000 complete [ 2892.772570] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.792940] LustreError: Skipped 1 previous similar message [ 2893.810100] LustreError: 103518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199201 with bad export cookie 4121715986294349840 [ 2893.815727] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2893.822637] LustreError: 103518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.156244] Lustre: server umount lustre-MDT0001 complete [ 2906.034953] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2908.348765] Key type lgssc unregistered [ 2908.642550] LNet: 107192:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2908.649136] LNet: Removed LNI 192.168.203.104@tcp [ 2909.451680] Key type .llcrypt unregistered [ 2909.456174] Key type ._llcrypt unregistered [ 2927.181921] alg: No test for adler32 (adler32-zlib) [ 2927.945385] Key type ._llcrypt registered [ 2927.947948] Key type .llcrypt registered [ 2928.038978] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2929.184580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2929.785729] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 2930.022098] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2930.026906] LNet: Accept secure, port 988 [ 2931.727234] Key type lgssc registered [ 2933.004422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.086550] LustreError: 108337:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2939.107610] LustreError: 108344:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2939.112196] LustreError: 108337:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4977 [ 2939.116149] LustreError: 108337:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2939.120850] LustreError: 108337:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2939.135459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.645540] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.699651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.017745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.420449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.774166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2951.839184] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2956.600224] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing lsmod [ 2958.976255] LustreError: 109647:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2958.996582] LustreError: 109652:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2959.007609] LustreError: 109647:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4974 [ 2959.012914] LustreError: 109647:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2959.025248] LustreError: 109647:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2959.029806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2959.433089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.535951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.540030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.551287] Lustre: Skipped 2 previous similar messages [ 2962.560704] Lustre: Skipped 2 previous similar messages [ 2967.739529] Lustre: server umount lustre-MDT0000 complete [ 2968.037968] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.050913] LustreError: Skipped 2 previous similar messages [ 2970.544176] LustreError: 109690:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199278 with bad export cookie 11619840030797049416 [ 2970.546438] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.552227] LustreError: 109690:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2970.721800] Lustre: server umount lustre-MDT0001 complete [ 2979.807735] Lustre: 110246:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199281/real 1776199281] req@000000003976c8c3 x1862479891146176/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199287 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2979.969269] Lustre: server umount lustre-OST0000 complete [ 2986.496751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.899617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.960845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.614503] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 2992.100198] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.353933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.710671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.535159] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3009.392613] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3013.943397] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3018.964514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.290620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.025400] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3029.260451] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3036.726494] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3046.899659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.901068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.901176] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3046.924484] Lustre: Skipped 1 previous similar message [ 3046.943142] Lustre: Skipped 1 previous similar message [ 3052.005648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3052.013285] Lustre: Skipped 1 previous similar message [ 3052.819122] Lustre: server umount lustre-OST0000 complete [ 3060.193425] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3060.201104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.219957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.240349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.768642] Lustre: server umount lustre-MDT0000 complete [ 3067.361386] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.383591] LustreError: Skipped 2 previous similar messages [ 3068.079101] LustreError: 110700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199375 with bad export cookie 11619840030797050116 [ 3068.080351] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.085630] LustreError: 110700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3082.376281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3085.695520] Key type lgssc unregistered [ 3085.964869] LNet: 114403:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3085.975389] LNet: Removed LNI 192.168.203.104@tcp [ 3086.552828] Key type .llcrypt unregistered [ 3086.555983] Key type ._llcrypt unregistered [ 3106.414067] alg: No test for adler32 (adler32-zlib) [ 3107.167805] Key type ._llcrypt registered [ 3107.172536] Key type .llcrypt registered [ 3107.276860] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3108.247166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3108.587302] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3108.753882] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3108.757568] LNet: Accept secure, port 988 [ 3110.408186] Key type lgssc registered [ 3111.224488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3118.222730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.695928] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.779127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.660596] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3125.218601] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.340710] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.779719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.187086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3136.822089] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3143.601936] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3149.136697] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.326756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.622382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.492945] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3166.210648] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3169.970746] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3174.663965] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3177.636555] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3182.566562] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.577167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.596820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3183.589335] Lustre: server umount lustre-OST0000 complete [ 3188.704094] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.712851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.727519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.754722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.757907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.791038] Lustre: Skipped 1 previous similar message [ 3193.311605] Lustre: 118823:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199494/real 1776199494] req@00000000933bdac8 x1862480078849024/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776199500 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3193.466629] Lustre: server umount lustre-MDT0000 complete [ 3195.877035] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.902714] LustreError: Skipped 1 previous similar message [ 3197.076722] LustreError: 115848:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199504 with bad export cookie 13490034659321164877 [ 3197.081452] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.090176] LustreError: 115848:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3197.444630] Lustre: server umount lustre-MDT0001 complete [ 3207.904787] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3209.973675] Key type lgssc unregistered [ 3210.199780] LNet: 119636:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3210.215923] LNet: Removed LNI 192.168.203.104@tcp [ 3210.869454] Key type .llcrypt unregistered [ 3210.873079] Key type ._llcrypt unregistered [ 3224.672918] alg: No test for adler32 (adler32-zlib) [ 3225.427837] Key type ._llcrypt registered [ 3225.431696] Key type .llcrypt registered [ 3225.564507] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3226.545259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3227.062176] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3227.257565] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3227.262369] LNet: Accept secure, port 988 [ 3228.927162] Key type lgssc registered [ 3230.034764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3237.882396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.334971] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.382463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.507884] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3244.514080] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.634499] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.824535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.044199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3254.565861] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3261.173455] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3266.932250] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3272.577970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.803244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.352456] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3284.466845] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3290.223557] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3300.838265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.838493] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.856593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.870648] Lustre: Skipped 1 previous similar message [ 3303.138618] Lustre: server umount lustre-OST0000 complete [ 3307.490761] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3307.505716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.524165] Lustre: Skipped 1 previous similar message [ 3307.528881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.072307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.075932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.193445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.200876] Lustre: Skipped 2 previous similar messages [ 3321.311128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3321.426492] Lustre: server umount lustre-MDT0000 complete [ 3322.341377] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3322.367267] LustreError: Skipped 1 previous similar message [ 3325.016409] LustreError: 120884:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199632 with bad export cookie 15699407007248149092 [ 3325.024896] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3325.026628] LustreError: 120884:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3325.264523] Lustre: server umount lustre-MDT0001 complete [ 3335.931882] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3338.535722] Key type lgssc unregistered [ 3338.829761] LNet: 124579:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.840877] LNet: Removed LNI 192.168.203.104@tcp [ 3339.602353] Key type .llcrypt unregistered [ 3339.609074] Key type ._llcrypt unregistered [ 3364.654415] alg: No test for adler32 (adler32-zlib) [ 3365.408719] Key type ._llcrypt registered [ 3365.413465] Key type .llcrypt registered [ 3365.554735] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3366.605316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3367.530883] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3367.846956] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3367.861563] LNet: Accept secure, port 988 [ 3369.538210] Key type lgssc registered [ 3370.581982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3378.843535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.323227] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.411667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.357842] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3385.830431] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.947594] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.157498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.508613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.953258] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3402.233583] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3408.114756] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3414.122870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.601484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.505419] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3427.645811] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3430.987936] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3438.562759] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3442.557472] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3442.561350] Lustre: Skipped 2 previous similar messages [ 3443.799710] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3443.865725] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3447.192275] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3447.202563] Lustre: Skipped 2 previous similar messages [ 3453.516895] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3453.526950] Lustre: Skipped 1 previous similar message [ 3457.833380] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3457.843146] Lustre: Skipped 2 previous similar messages [ 3462.758487] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp [ 3473.898511] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3473.911064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.938560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3474.651454] Lustre: server umount lustre-OST0000 complete [ 3479.520842] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.526544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.536124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.335251] Lustre: 125374:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199782/real 1776199782] req@00000000ea4dbfeb x1862480350440768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776199789 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3482.369428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.381468] Lustre: Skipped 1 previous similar message [ 3482.385748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.390575] Lustre: Skipped 3 previous similar messages [ 3484.127152] Lustre: 125373:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199784/real 1776199784] req@0000000081b6b456 x1862480350440896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776199791 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3484.483181] Lustre: server umount lustre-MDT0000 complete [ 3487.199511] Lustre: 125375:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199787/real 1776199787] req@0000000077eb1a2f x1862480350441664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776199794 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3487.714473] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.734112] LustreError: Skipped 1 previous similar message [ 3488.296282] LustreError: 129171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199795 with bad export cookie 14146927239943083863 [ 3488.302918] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3488.305500] LustreError: 129171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3488.705847] Lustre: server umount lustre-MDT0001 complete [ 3500.472258] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3503.218666] Key type lgssc unregistered [ 3503.464792] LNet: 130828:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3503.478382] LNet: Removed LNI 192.168.203.104@tcp [ 3504.187853] Key type .llcrypt unregistered [ 3504.190133] Key type ._llcrypt unregistered [ 3527.072439] alg: No test for adler32 (adler32-zlib) [ 3527.840371] Key type ._llcrypt registered [ 3527.843962] Key type .llcrypt registered [ 3527.982273] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3528.954536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3529.748286] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3529.957866] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3529.961598] LNet: Accept secure, port 988 [ 3531.631266] Key type lgssc registered [ 3532.667660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3540.655562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.093662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3542.139623] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3542.183812] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp [ 3542.186658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.557578] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3547.616721] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.736084] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.109426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.461534] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3554.491590] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp [ 3554.559427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.824531] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3566.055844] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3571.017635] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3576.486744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.779143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.861241] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3586.940467] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.169022] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3608.554846] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3608.562691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.585867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3608.588279] Lustre: Skipped 1 previous similar message [ 3610.198283] Lustre: server umount lustre-OST0000 complete [ 3613.665365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.672214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.680837] Lustre: Skipped 2 previous similar messages [ 3613.699793] Lustre: Skipped 1 previous similar message [ 3618.784955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.801420] Lustre: Skipped 1 previous similar message [ 3619.818307] Lustre: server umount lustre-MDT0000 complete [ 3623.388588] LustreError: 132418:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199930 with bad export cookie 11373177394227276496 [ 3623.393345] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.402617] LustreError: 132418:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3623.651347] Lustre: server umount lustre-MDT0001 complete [ 3633.704709] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3636.269637] Key type lgssc unregistered [ 3636.513896] LNet: 136165:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3636.520510] LNet: Removed LNI 192.168.203.104@tcp [ 3637.192399] Key type .llcrypt unregistered [ 3637.194925] Key type ._llcrypt unregistered [ 3660.706822] alg: No test for adler32 (adler32-zlib) [ 3661.462981] Key type ._llcrypt registered [ 3661.468565] Key type .llcrypt registered [ 3661.572993] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3673.148963] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3673.980810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3674.407554] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3674.577935] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3674.585451] LNet: Accept secure, port 988 [ 3676.295213] Key type lgssc registered [ 3677.096181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3683.945977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.306501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3695.071346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3701.133843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.221974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.291637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.558945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3708.591952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3708.678277] Lustre: lustre-MDT0000: new disk, initializing [ 3708.752278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.771553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3712.993377] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3722.599297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3722.688942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.815366] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3722.860469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3722.867370] Lustre: Skipped 1 previous similar message [ 3722.952993] Lustre: lustre-MDT0001: new disk, initializing [ 3723.048944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3723.093595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3723.107651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3726.999896] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3736.065378] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3741.147394] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3747.796645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.868158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.879363] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3748.097614] Lustre: lustre-OST0000: new disk, initializing [ 3748.101601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3748.172456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.140857] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3755.558148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3755.569213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3762.246901] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3767.161673] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3767.428497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3772.396448] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3772.630588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3775.967820] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3775.973128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.987273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.485551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3776.487778] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3776.499712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.499719] Lustre: Skipped 1 previous similar message [ 3780.723987] Lustre: server umount lustre-OST0000 complete [ 3784.672583] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3784.688104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.698115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.731640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.731981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.729632] Lustre: server umount lustre-MDT0000 complete [ 3791.844871] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.869937] LustreError: Skipped 1 previous similar message [ 3794.623728] LustreError: 139681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200102 with bad export cookie 9541585044899407717 [ 3794.633606] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.656480] LustreError: 139681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3794.971715] Lustre: server umount lustre-MDT0001 complete [ 3804.793820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.137918] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.201870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.273552] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.884849] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3815.394835] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.515850] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.949204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.323418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.674819] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3836.173308] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3843.723964] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3850.852463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.167499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.683905] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3864.300790] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3869.055669] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3869.338320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3874.123666] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3874.399055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3882.074675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3882.126726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3882.135182] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3882.275527] Lustre: lustre-OST0001: new disk, initializing [ 3882.280732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3882.370429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3887.291618] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 3890.345888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3890.363913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3896.498651] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3896.814841] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3901.739708] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3902.005087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3908.633822] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98d707119800.ost_server_uuid 40 [ 3910.605213] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98d707119800.ost_server_uuid in IDLE state after 0 sec [ 3921.384426] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3921.392531] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.397581] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3921.397590] Lustre: Skipped 1 previous similar message [ 3921.408810] LustreError: Skipped 1 previous similar message [ 3921.453135] Lustre: Skipped 1 previous similar message [ 3922.169762] Lustre: server umount lustre-OST0001 complete [ 3926.499984] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.516228] LustreError: Skipped 1 previous similar message [ 3931.617700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.630928] Lustre: Skipped 4 previous similar messages [ 3940.326154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3940.485441] Lustre: server umount lustre-OST0000 complete [ 3946.977937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.996495] Lustre: Skipped 3 previous similar messages [ 3950.776366] Lustre: server umount lustre-MDT0000 complete [ 3952.098140] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.115639] LustreError: Skipped 5 previous similar messages [ 3954.315324] LustreError: 144027:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200261 with bad export cookie 9541585044899408571 [ 3954.320058] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3954.325951] LustreError: 144027:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3974.355141] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3977.010539] Key type lgssc unregistered [ 3977.238511] LNet: 150699:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3977.248537] LNet: Removed LNI 192.168.203.104@tcp [ 3977.952763] Key type .llcrypt unregistered [ 3977.954487] Key type ._llcrypt unregistered [ 4007.790368] alg: No test for adler32 (adler32-zlib) [ 4008.546502] Key type ._llcrypt registered [ 4008.548086] Key type .llcrypt registered [ 4008.654650] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4021.149737] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4022.399188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.886688] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4023.177918] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4023.189192] LNet: Accept secure, port 988 [ 4024.855966] Key type lgssc registered [ 4026.151565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4036.259910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4042.461671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.647842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4053.717608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4060.459356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4060.562992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.872612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4061.902534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4062.026326] Lustre: lustre-MDT0000: new disk, initializing [ 4062.145345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.176043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4067.837973] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4080.150905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4080.282367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.570877] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4080.654735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4080.674236] Lustre: Skipped 1 previous similar message [ 4080.809668] Lustre: lustre-MDT0001: new disk, initializing [ 4080.960668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4081.017605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4081.033894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4086.810316] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4102.413171] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4110.668936] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4119.080256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4119.166194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4119.172513] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4119.422676] Lustre: lustre-OST0000: new disk, initializing [ 4119.431072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4119.530728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.104891] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4126.792613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4126.806365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4133.380884] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4137.685770] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4137.866824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4142.165712] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4142.381420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4147.683507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.685754] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4147.693980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.709610] Lustre: Skipped 1 previous similar message [ 4147.730295] LustreError: Skipped 1 previous similar message [ 4150.498967] Lustre: server umount lustre-OST0000 complete [ 4157.920042] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4157.923506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.940307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.940316] Lustre: Skipped 1 previous similar message [ 4157.981142] Lustre: Skipped 1 previous similar message [ 4160.643644] Lustre: server umount lustre-MDT0000 complete [ 4163.049214] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.059924] LustreError: Skipped 1 previous similar message [ 4164.080087] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.106495] LustreError: Skipped 1 previous similar message [ 4164.792951] LustreError: 154792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200472 with bad export cookie 10889631803240245128 [ 4164.808428] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4165.197162] Lustre: server umount lustre-MDT0001 complete [ 4175.156854] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 17:01:20 (1776200480) [ 4193.763412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4204.870672] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4215.561586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4221.032676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4226.232649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4231.402883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4237.595612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4237.657277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.872691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4237.910074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4238.010281] Lustre: lustre-MDT0000: new disk, initializing [ 4238.113762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.137800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4242.425954] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4252.542606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4252.610565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.684218] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4252.727823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4252.732917] Lustre: Skipped 1 previous similar message [ 4252.816571] Lustre: lustre-MDT0001: new disk, initializing [ 4252.925491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4252.969910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4252.978935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4257.873824] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4268.073041] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4274.471817] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4281.421697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4281.514378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.522529] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4281.852103] Lustre: lustre-OST0000: new disk, initializing [ 4281.870828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4281.976993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.603789] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4288.053652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4288.064322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4295.280969] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4299.080831] Lustre: Failing over lustre-OST0000 [ 4299.219502] Lustre: server umount lustre-OST0000 complete [ 4308.895179] Lustre: 153019:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200609/real 1776200609] req@0000000053a13b07 x1862481037266496/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1776200616 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4308.916752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4311.007126] Lustre: 153019:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200611/real 1776200611] req@000000007f94a690 x1862481037266944/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776200618 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4311.519854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.615197] Lustre: 153018:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200616/real 1776200616] req@000000003d13637d x1862481037267584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776200623 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4315.658544] Lustre: 153018:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4318.063482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.387712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.404912] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4322.199250] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4325.501890] Lustre: Failing over lustre-MDT0000 [ 4325.729945] Lustre: server umount lustre-MDT0000 complete [ 4325.869404] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4326.922875] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4327.904853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.911190] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.923775] Lustre: Skipped 1 previous similar message [ 4327.947806] LustreError: Skipped 3 previous similar messages [ 4331.984040] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.000421] LustreError: Skipped 3 previous similar messages [ 4333.535145] Lustre: 161804:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200634/real 1776200634] req@00000000287c2b6b x1862481037272960/t0(0) o101->MGC192.168.203.104@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1776200641 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4333.576711] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.285186] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.296696] LustreError: Skipped 15 previous similar messages [ 4345.029122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.952574] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x971fbe6de6ee6ade to 0x971fbe6de6ee6f99 [ 4350.972818] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 4351.196150] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4351.239982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.329740] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4355.540338] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4356.542846] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4356.613785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4356.656338] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4356.705974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4366.819644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.820941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4366.840755] Lustre: Skipped 2 previous similar messages [ 4366.851851] Lustre: Skipped 1 previous similar message [ 4371.348958] Lustre: server umount lustre-OST0000 complete [ 4377.063619] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4377.070579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.088592] Lustre: Skipped 2 previous similar messages [ 4382.176319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4382.198326] Lustre: Skipped 1 previous similar message [ 4389.343131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4389.519574] Lustre: server umount lustre-MDT0000 complete [ 4393.147854] LustreError: 161793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200700 with bad export cookie 10889631803240247193 [ 4393.156705] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4393.161696] LustreError: 161793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4404.617924] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4407.605217] Key type lgssc unregistered [ 4407.916541] LNet: 167657:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4407.930484] LNet: Removed LNI 192.168.203.104@tcp [ 4408.721515] Key type .llcrypt unregistered [ 4408.727767] Key type ._llcrypt unregistered [ 4430.316468] alg: No test for adler32 (adler32-zlib) [ 4431.074876] Key type ._llcrypt registered [ 4431.079142] Key type .llcrypt registered [ 4431.186066] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4432.216765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4432.811705] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4433.035416] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4433.040548] LNet: Accept secure, port 988 [ 4434.738203] Key type lgssc registered [ 4435.793633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4444.982939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.330243] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.385135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.809577] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.426683] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4456.931034] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.052502] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.468309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.934928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4468.602341] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4476.322296] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4482.436724] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4488.618663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.907526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.091975] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 4497.091778] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4500.664141] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4518.491753] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4525.607459] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4537.355665] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4626.981362] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4773.378665] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4842.984546] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4949.500592] Lustre: lustre-MDT0000: Client a51681bb-77cf-4bd4-a7a9-41e2b37ff694 (at 192.168.203.4@tcp) reconnecting [ 4981.613799] Lustre: Failing over lustre-MDT0000 [ 4981.770048] Lustre: server umount lustre-MDT0000 complete [ 4983.777513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.782172] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4983.796422] Lustre: Skipped 2 previous similar messages [ 4983.827490] LustreError: Skipped 2 previous similar messages [ 4994.019600] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.038182] LustreError: Skipped 7 previous similar messages [ 5000.818810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.028092] LustreError: 11-0: MGC192.168.203.104@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 5001.033418] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5001.046908] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x3f3a85af860ee17b to 0x3f3a85af86114da3 [ 5001.047118] LustreError: 168434:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 5001.073361] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 5001.373053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.443493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5005.778654] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5006.311572] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5006.821523] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5006.866024] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5006.902395] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 5016.211942] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5017.526755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5037.544757] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.549021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.558679] Lustre: Skipped 2 previous similar messages [ 5037.558954] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5037.587937] Lustre: Skipped 2 previous similar messages [ 5040.876509] Lustre: server umount lustre-MDT0000 complete [ 5042.658601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.681665] LustreError: Skipped 10 previous similar messages [ 5044.333234] LustreError: 169103:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201351 with bad export cookie 4556100961953729955 [ 5044.335120] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5044.343344] LustreError: 169103:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5047.782431] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.805177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5049.827327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5050.657800] Lustre: server umount lustre-MDT0001 complete [ 5068.255353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5068.429763] Lustre: server umount lustre-OST0000 complete [ 5076.663899] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5084.411557] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5091.972749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5098.031886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5103.850996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5109.482324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5115.534938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5115.598698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.836290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5115.857664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5115.945615] Lustre: lustre-MDT0000: new disk, initializing [ 5116.037796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.056658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5119.784492] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5128.934870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5128.995506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.104265] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5129.181798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5129.186106] Lustre: Skipped 1 previous similar message [ 5129.275577] Lustre: lustre-MDT0001: new disk, initializing [ 5129.374520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5129.417224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5129.427252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5133.450564] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5142.407980] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5147.967644] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.131947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.192845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.200674] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5154.445946] Lustre: lustre-OST0000: new disk, initializing [ 5154.450300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5154.551841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.750596] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5160.063723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5160.070458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5167.633498] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5172.026937] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5172.227030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5176.635807] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5176.841452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5180.896674] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5180.897462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.909807] LustreError: Skipped 1 previous similar message [ 5180.926832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5180.935828] Lustre: Skipped 1 previous similar message [ 5184.658868] Lustre: server umount lustre-OST0000 complete [ 5191.138077] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5191.147126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.155985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.171744] Lustre: Skipped 2 previous similar messages [ 5194.492485] Lustre: server umount lustre-MDT0000 complete [ 5196.257048] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.272450] LustreError: Skipped 3 previous similar messages [ 5198.228940] LustreError: 178128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201505 with bad export cookie 4556100961953730676 [ 5198.234127] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.243833] LustreError: 178128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5207.151871] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:18:33 (1776201513) [ 5225.828632] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5236.079817] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5246.795601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5253.269389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5259.079428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5264.759506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.729810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.805975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.990785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5273.055457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5273.161367] Lustre: lustre-MDT0000: new disk, initializing [ 5273.255305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.278112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5277.172828] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5287.135563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5287.193481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.269595] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5287.321859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5287.326093] Lustre: Skipped 1 previous similar message [ 5287.381323] Lustre: lustre-MDT0001: new disk, initializing [ 5287.476935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5287.483297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5291.530680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5300.554240] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5307.378714] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5313.448647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5313.525928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5313.532313] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5313.771377] Lustre: lustre-OST0000: new disk, initializing [ 5313.777925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5317.690564] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5323.873825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5323.895459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5326.044757] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5334.503246] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5334.503422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.526175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.538776] Lustre: Skipped 2 previous similar messages [ 5338.259488] Lustre: server umount lustre-OST0000 complete [ 5338.262555] Lustre: Skipped 1 previous similar message [ 5343.712984] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5343.722673] LustreError: Skipped 1 previous similar message [ 5349.857131] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.872121] LustreError: Skipped 3 previous similar messages [ 5351.459507] LustreError: 183025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201658 with bad export cookie 4556100961953731530 [ 5351.466492] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5351.482908] LustreError: 183025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5358.398501] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:21:04 (1776201664) [ 5371.003410] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5377.916892] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5387.197167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5394.106425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.803571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.186363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.357338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.433508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.657359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5412.683459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5412.793621] Lustre: lustre-MDT0000: new disk, initializing [ 5412.871360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5412.874251] Lustre: Skipped 2 previous similar messages [ 5412.886821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5416.707060] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5426.217726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5426.281965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.380309] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5426.634888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5431.649927] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5441.104590] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5446.960614] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5454.223152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5454.339645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5454.346251] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5454.675923] Lustre: lustre-OST0000: new disk, initializing [ 5454.677519] Lustre: Skipped 1 previous similar message [ 5454.682820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5454.702055] Lustre: Skipped 2 previous similar messages [ 5459.509394] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5462.620498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5462.633447] Lustre: Skipped 1 previous similar message [ 5462.648779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5468.966866] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5478.370313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5478.373128] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5478.386546] Lustre: Skipped 3 previous similar messages [ 5478.396394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5478.427701] Lustre: Skipped 4 previous similar messages [ 5481.318549] Lustre: server umount lustre-OST0000 complete [ 5481.321881] Lustre: Skipped 2 previous similar messages [ 5493.728557] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.744328] LustreError: Skipped 3 previous similar messages [ 5495.284671] LustreError: 190355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201802 with bad export cookie 4556100961953732615 [ 5495.288339] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5495.296076] LustreError: 190355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5507.330091] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5509.631449] Key type lgssc unregistered [ 5509.935288] LNet: 193459:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5509.953538] LNet: Removed LNI 192.168.203.104@tcp [ 5510.846332] Key type .llcrypt unregistered [ 5510.850200] Key type ._llcrypt unregistered [ 5530.570608] alg: No test for adler32 (adler32-zlib) [ 5531.332665] Key type ._llcrypt registered [ 5531.334784] Key type .llcrypt registered [ 5531.456582] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5532.407066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5532.836926] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 5533.074242] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5533.086732] LNet: Accept secure, port 988 [ 5534.783134] Key type lgssc registered [ 5535.611434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5542.259326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.593766] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.634363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.014916] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5549.028957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.146295] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.042474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.393513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.914463] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5566.774110] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.556862] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5579.310320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5579.634277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.938826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5590.642955] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5593.970346] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5614.572992] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5614.574689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.585806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5614.619670] Lustre: Skipped 1 previous similar message [ 5617.759158] Lustre: server umount lustre-OST0000 complete [ 5622.242768] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5622.258181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.274662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5622.278359] Lustre: Skipped 1 previous similar message [ 5624.803323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.804975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.615775] Lustre: server umount lustre-MDT0000 complete [ 5629.926633] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.944439] LustreError: Skipped 1 previous similar message [ 5631.218582] LustreError: 194902:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201938 with bad export cookie 12569163584037429762 [ 5631.241152] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5631.594430] Lustre: server umount lustre-MDT0001 complete [ 5642.977906] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5645.728536] Key type lgssc unregistered [ 5646.027586] LNet: 198600:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5646.046468] LNet: Removed LNI 192.168.203.104@tcp [ 5646.906407] Key type .llcrypt unregistered [ 5646.909868] Key type ._llcrypt unregistered [ 5669.891808] alg: No test for adler32 (adler32-zlib) [ 5670.655214] Key type ._llcrypt registered [ 5670.658882] Key type .llcrypt registered [ 5670.813649] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5672.015186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5672.490918] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 5672.669612] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5672.673713] LNet: Accept secure, port 988 [ 5674.399138] Key type lgssc registered [ 5675.659952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5683.607385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.040588] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.083905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.401889] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5690.343628] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.461926] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.110562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.495096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.332253] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5708.208442] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5713.495686] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5720.194798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5720.482400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.293940] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5731.716624] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5734.843818] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5742.576237] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 5745.122098] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.124175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.142370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.151099] Lustre: Skipped 1 previous similar message [ 5745.185038] Lustre: Skipped 1 previous similar message [ 5747.488696] Lustre: server umount lustre-OST0000 complete [ 5755.027643] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98d70324e800.ost_server_uuid 40 [ 5756.897727] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5774.306812] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5774.327380] LustreError: Skipped 1 previous similar message [ 5774.330860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.350936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.864706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.866361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.716532] Lustre: server umount lustre-MDT0000 complete [ 5782.827361] LustreError: 200045:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202090 with bad export cookie 16190935545658995133 [ 5782.829389] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.848962] LustreError: 200045:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5783.331342] Lustre: server umount lustre-MDT0001 complete [ 5791.676713] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:28:17 (1776202097) [ 5798.930133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.686965] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.800967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.025555] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.339520] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5815.055087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.265648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5815.274538] Lustre: Skipped 3 previous similar messages [ 5815.388321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.507089] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5830.007787] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5837.174563] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.751784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5844.966976] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.981690] LustreError: Skipped 1 previous similar message [ 5845.148909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.125733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5850.245106] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5858.497889] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5866.052589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5866.146165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5866.154474] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5866.274634] Lustre: lustre-OST0001: new disk, initializing [ 5866.282600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5866.399632] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5870.626591] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5874.332894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5874.354579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5879.332184] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5884.904307] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 5884.910948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.931988] Lustre: Skipped 1 previous similar message [ 5887.424611] Lustre: server umount lustre-OST0000 complete [ 5890.003479] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.672443] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5893.903577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5898.995989] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5899.323413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5915.634976] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5915.642763] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.658625] LustreError: Skipped 1 previous similar message [ 5915.660846] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5915.660854] Lustre: Skipped 2 previous similar messages [ 5915.700230] Lustre: Skipped 1 previous similar message [ 5920.440295] Lustre: server umount lustre-OST0001 complete [ 5925.857847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.877800] Lustre: Skipped 1 previous similar message [ 5931.484733] Lustre: server umount lustre-MDT0000 complete [ 5932.011932] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.038312] LustreError: Skipped 17 previous similar messages [ 5935.470366] LustreError: 206967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202242 with bad export cookie 16190935545658996197 [ 5935.483980] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5935.485668] LustreError: 206967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5941.384440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5945.815507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5949.550040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5953.350053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.036434] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:31:05 (1776202265) [ 5964.683940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.032257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5965.066554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5965.236788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.867267] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5977.331976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.396322] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5977.439582] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5981.483725] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 5988.001231] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5992.523231] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5999.166714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5999.364495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6003.810490] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6009.485712] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 6010.822227] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6017.872408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6017.966452] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6023.004220] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6027.494442] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 6032.831221] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6037.988440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.020104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6038.032783] Lustre: Skipped 6 previous similar messages [ 6042.633697] Lustre: server umount lustre-OST0000 complete [ 6042.639502] Lustre: Skipped 1 previous similar message [ 6043.109091] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.158924] LustreError: Skipped 1 previous similar message [ 6063.592098] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6063.603515] LustreError: Skipped 1 previous similar message [ 6069.731917] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6073.829187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.845690] Lustre: Skipped 4 previous similar messages [ 6075.517559] Lustre: server umount lustre-MDT0000 complete [ 6075.520898] Lustre: Skipped 1 previous similar message [ 6078.876840] LustreError: 210478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202386 with bad export cookie 16190935545658997583 [ 6078.878602] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.892735] LustreError: 210478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6085.427874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6089.416574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6093.301118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6097.306736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6102.912873] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:33:28 (1776202408) [ 6118.092598] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6125.732205] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6133.844537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6139.353139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6144.686252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6150.627881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6157.982285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6158.059495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.265619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6158.292430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6158.349676] Lustre: lustre-MDT0000: new disk, initializing [ 6158.411353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6158.418028] Lustre: Skipped 3 previous similar messages [ 6158.429671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6163.820557] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6174.044139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6174.173638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.337967] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6174.396883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6174.405287] Lustre: Skipped 1 previous similar message [ 6174.483360] Lustre: lustre-MDT0001: new disk, initializing [ 6174.606152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6174.622022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6179.059927] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6187.976523] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6192.980255] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6198.045624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6198.095597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.101330] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6198.293187] Lustre: lustre-OST0000: new disk, initializing [ 6198.297889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6201.885753] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6204.491842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6204.506140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6210.156301] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6214.257755] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6214.455396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6218.179384] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6218.370626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6220.268456] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6220.271295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.306167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6220.315796] Lustre: Skipped 7 previous similar messages [ 6226.314018] Lustre: server umount lustre-OST0000 complete [ 6226.322860] Lustre: Skipped 1 previous similar message [ 6239.144863] LustreError: 218713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202546 with bad export cookie 16190935545658999319 [ 6239.148657] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6239.153031] LustreError: 218713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6247.538112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.035463] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.049788] LustreError: Skipped 11 previous similar messages [ 6252.192322] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6259.406429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.239326] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6272.561917] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6279.283964] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6286.245379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.526409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.534976] Lustre: Skipped 4 previous similar messages [ 6290.714736] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6297.620106] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6301.870974] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6302.050912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6305.573807] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6305.781462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6309.350091] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6309.365098] LustreError: Skipped 1 previous similar message [ 6319.635654] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6323.213711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6350.985510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6355.807556] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6361.511686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.522956] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.528101] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 6361.542261] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 6361.555261] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6363.537994] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6365.069055] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6371.808793] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.818824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.831212] Lustre: Skipped 5 previous similar messages [ 6371.836505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.840451] Lustre: Skipped 9 previous similar messages [ 6375.179380] Lustre: server umount lustre-OST0000 complete [ 6375.184422] Lustre: Skipped 3 previous similar messages [ 6388.581415] LustreError: 228402:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202696 with bad export cookie 16190935545659000173 [ 6388.583015] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.591440] LustreError: 228402:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6396.805541] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:38:22 (1776202702) [ 6402.473187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.715628] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6414.075387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.448651] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6425.833859] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6432.209674] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6438.850504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6444.020839] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6452.943744] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6458.343358] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6458.555923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6464.155324] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6464.430619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6471.454884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6471.573922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6471.582311] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6471.815994] Lustre: lustre-OST0001: new disk, initializing [ 6471.838827] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6476.700948] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6481.512115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6481.522214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6485.470699] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6489.858576] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6490.178660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6495.447547] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6495.721765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6502.372460] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.380293] LustreError: Skipped 2 previous similar messages [ 6507.499524] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.519174] LustreError: Skipped 10 previous similar messages [ 6509.524100] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6509.782997] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6513.992976] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6514.165158] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6542.215976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.663818] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6547.950341] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6547.970911] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6547.975641] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6547.998108] Lustre: Skipped 2 previous similar messages [ 6553.548226] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6555.024839] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6597.087342] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6610.344283] LustreError: 236501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202917 with bad export cookie 16190935545659001580 [ 6610.348111] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.354400] LustreError: 236501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6621.859139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6625.437302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6629.146687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6632.660662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6638.356979] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:42:24 (1776202944) [ 6643.766152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.039883] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6644.084213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6644.240784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6644.248948] Lustre: Skipped 6 previous similar messages [ 6648.465152] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6656.211990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.311842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6656.336349] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6660.363893] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6667.377076] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6673.262767] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6678.582667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6678.712890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6682.455459] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6689.336843] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6697.131118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6697.216672] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6701.336534] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6708.463887] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6709.986025] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6715.438032] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6715.748866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6720.126378] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6720.331632] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6726.202042] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98d7040e5000.ost_server_uuid 40 [ 6727.981433] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98d7040e5000.ost_server_uuid in IDLE state after 0 sec [ 6749.516603] Lustre: Permanently deactivating lustre-OST0001 [ 6749.532112] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6753.166568] Lustre: Permanently reactivating lustre-OST0001 [ 6761.953434] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.973219] Lustre: Skipped 11 previous similar messages [ 6761.990163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6762.000268] Lustre: Skipped 19 previous similar messages [ 6763.449166] Lustre: server umount lustre-OST0001 complete [ 6763.451389] Lustre: Skipped 7 previous similar messages [ 6769.632075] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6769.651448] LustreError: Skipped 4 previous similar messages [ 6791.135146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6794.542379] LustreError: 238185:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203102 with bad export cookie 16190935545659002861 [ 6794.548759] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6794.562050] LustreError: 238185:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6800.851765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.158201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6809.173230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6813.290829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6819.913043] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:45:26 (1776203126) [ 6821.162337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.116829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.558856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6828.584312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6828.589382] Lustre: Skipped 5 previous similar messages [ 6833.398890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6841.656956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6841.774426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6847.513237] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6854.898697] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6860.173474] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6866.682537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6866.757476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6866.934216] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6866.961988] Lustre: Permanently deactivating lustre-OST0000 [ 6871.291969] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6875.666977] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6878.828188] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6885.025325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6885.116725] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6889.088806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 6896.357845] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6898.669612] Lustre: Permanently reactivating lustre-OST0000 [ 6898.694420] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6898.702115] Lustre: Skipped 4 previous similar messages [ 6900.766530] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6900.772940] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6900.790594] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6900.840266] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6900.861687] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 6941.151750] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6944.715420] LustreError: 248010:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203252 with bad export cookie 16190935545659004485 [ 6944.716676] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.730887] LustreError: 248010:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6956.209940] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6959.053915] Key type lgssc unregistered [ 6959.354569] LNet: 250025:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6959.365684] LNet: Removed LNI 192.168.203.104@tcp [ 6960.410631] Key type .llcrypt unregistered [ 6960.421590] Key type ._llcrypt unregistered [ 6983.151085] alg: No test for adler32 (adler32-zlib) [ 6983.907199] Key type ._llcrypt registered [ 6983.909969] Key type .llcrypt registered [ 6984.046524] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6985.130046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6985.463938] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 6985.617491] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6985.622739] LNet: Accept secure, port 988 [ 6987.279141] Key type lgssc registered [ 6988.126826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6991.384201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6997.565180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.986719] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.069563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.656054] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7004.134940] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.254050] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.187557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7012.279678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.447810] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7012.504699] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7012.512858] Lustre: Permanently deactivating lustre-MDT0001 [ 7012.583993] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7012.591907] LustreError: 251569:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7012.612811] LustreError: 251569:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7012.622197] LustreError: 251569:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 7012.917294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7016.801285] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7024.628138] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7031.053691] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7038.280933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.580191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.669664] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.699043] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 7043.365893] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7048.161971] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.202718] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7050.523283] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7057.264490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.511397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7061.943548] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7062.526597] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 7069.265847] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7072.250379] Lustre: Permanently reactivating lustre-MDT0001 [ 7072.265975] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7072.269804] Lustre: Skipped 3 previous similar messages [ 7077.901410] Lustre: lustre-MDT0001: Client a39f1a70-c4b2-478b-9b8a-3517d2a82cb4 (at 192.168.203.4@tcp) reconnecting [ 7084.702658] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7086.134600] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7092.050708] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7092.275658] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7096.942826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7097.177925] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7099.415246] Lustre: Permanently deactivating lustre-MDT0001 [ 7099.442760] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7099.445533] Lustre: Skipped 2 previous similar messages [ 7109.207537] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7118.817194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.820629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.844283] Lustre: Skipped 3 previous similar messages [ 7118.854052] Lustre: Skipped 3 previous similar messages [ 7123.210771] Lustre: server umount lustre-MDT0000 complete [ 7123.937809] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.962255] LustreError: Skipped 7 previous similar messages [ 7126.741726] LustreError: 251520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203434 with bad export cookie 13097259129649514893 [ 7126.745621] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.749030] LustreError: 251520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7127.083863] Lustre: server umount lustre-MDT0001 complete [ 7136.564523] Lustre: server umount lustre-OST0000 complete [ 7146.810456] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:50:53 (1776203453) [ 7157.283614] Lustre: server umount lustre-OST0001 complete [ 7164.437068] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7172.661417] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7182.514945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7188.650438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7194.788335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7201.415994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7208.537500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7208.616137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.821770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7208.826423] Lustre: Skipped 2 previous similar messages [ 7208.847359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7208.909592] Lustre: lustre-MDT0000: new disk, initializing [ 7209.000790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7209.015304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7213.089491] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7223.105366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7223.170601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.232936] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7223.270742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7223.275963] Lustre: Skipped 1 previous similar message [ 7223.347741] Lustre: lustre-MDT0001: new disk, initializing [ 7223.445745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7223.489119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7223.519633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7229.517366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7240.721448] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7245.446073] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7251.061838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7251.148994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7251.155531] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7251.359371] Lustre: lustre-OST0000: new disk, initializing [ 7251.365379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7251.445968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.134551] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7257.164935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7257.178173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7263.883485] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7265.514144] LustreError: 262128:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7274.289550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7274.431442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.444784] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7274.553431] Lustre: lustre-OST0001: new disk, initializing [ 7274.557563] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7275.533585] LustreError: 262128:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7275.563238] LustreError: 262128:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7279.897325] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7283.835557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7283.851264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7285.599591] LustreError: 262128:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7290.955929] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7292.950727] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7297.018792] Lustre: lustre-OST0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 7299.552050] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7299.552772] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7299.564598] LustreError: Skipped 1 previous similar message [ 7299.575229] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7299.579292] Lustre: Skipped 1 previous similar message [ 7301.522752] Lustre: server umount lustre-OST0001 complete [ 7302.093917] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.008722] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.023347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.046093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.058858] Lustre: Skipped 2 previous similar messages [ 7323.730599] Lustre: server umount lustre-OST0000 complete [ 7330.275644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.279473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.292046] Lustre: Skipped 2 previous similar messages [ 7330.310053] Lustre: Skipped 3 previous similar messages [ 7335.394222] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.405291] LustreError: Skipped 9 previous similar messages [ 7336.770620] LustreError: 260029:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203644 with bad export cookie 13097259129649517441 [ 7336.776067] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.782895] LustreError: 260029:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7351.263285] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7351.588651] Lustre: server umount lustre-MDT0001 complete [ 7351.593493] Lustre: Skipped 1 previous similar message [ 7363.230516] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7365.999426] Key type lgssc unregistered [ 7366.301564] LNet: 265241:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7366.315174] LNet: Removed LNI 192.168.203.104@tcp [ 7367.072705] Key type .llcrypt unregistered [ 7367.075639] Key type ._llcrypt unregistered [ 7399.524955] alg: No test for adler32 (adler32-zlib) [ 7400.281297] Key type ._llcrypt registered [ 7400.284263] Key type .llcrypt registered [ 7400.390917] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7413.066523] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7414.092241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7414.746832] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 7415.125970] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7415.136881] LNet: Accept secure, port 988 [ 7416.855120] Key type lgssc registered [ 7417.945662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7426.636120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7432.426847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7438.590235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7444.862149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7452.103206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7452.206360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.568144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7453.614198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7453.741416] Lustre: lustre-MDT0000: new disk, initializing [ 7453.920929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.977284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7458.672523] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7468.981846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7469.133941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.293917] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7469.341740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7469.348758] Lustre: Skipped 1 previous similar message [ 7469.516384] Lustre: lustre-MDT0001: new disk, initializing [ 7469.669900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7469.720482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7469.738633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7474.108103] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7482.228100] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7487.645805] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7495.320386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7495.420504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.438794] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7495.773603] Lustre: lustre-OST0000: new disk, initializing [ 7495.780296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7495.888119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7500.287539] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7505.525056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7505.537397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7510.084751] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7515.261921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7515.488095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7520.677867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7520.963849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7526.369078] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7526.373080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.378856] LustreError: Skipped 1 previous similar message [ 7526.381574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7526.401428] Lustre: Skipped 1 previous similar message [ 7526.420703] Lustre: Skipped 1 previous similar message [ 7528.874716] Lustre: server umount lustre-OST0000 complete [ 7536.098188] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7536.109745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7536.139095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7541.728215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7541.735300] Lustre: Skipped 2 previous similar messages [ 7546.857810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7546.866269] Lustre: Skipped 1 previous similar message [ 7547.361531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7547.507813] Lustre: server umount lustre-MDT0000 complete [ 7550.956330] LustreError: 270192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203858 with bad export cookie 12734774908413503706 [ 7550.965125] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.968282] LustreError: 270192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7551.251921] Lustre: server umount lustre-MDT0001 complete [ 7559.128858] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 17:57:45 (1776203865) [ 7564.762569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.298033] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.400480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.998695] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7570.402416] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.521864] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.241880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.658970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7583.236474] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7591.182724] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7597.131498] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7603.432976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7603.781122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.951942] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7616.183459] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7634.401373] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7634.414176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.433539] Lustre: Skipped 1 previous similar message [ 7634.443816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7639.593944] Lustre: server umount lustre-OST0000 complete [ 7644.052938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7656.973420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.433294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.524583] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7667.629463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7667.658683] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 7667.674056] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7668.936557] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7677.924632] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7677.948717] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 7677.955313] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7683.041957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.043379] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7683.067196] Lustre: Skipped 1 previous similar message [ 7683.070628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7683.074992] LustreError: Skipped 2 previous similar messages [ 7683.090763] Lustre: Skipped 3 previous similar messages [ 7686.403797] Lustre: server umount lustre-OST0000 complete [ 7691.232102] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7691.244143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.257746] Lustre: Skipped 1 previous similar message [ 7695.913696] Lustre: server umount lustre-MDT0000 complete [ 7698.401570] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.418337] LustreError: Skipped 1 previous similar message [ 7699.762876] LustreError: 273875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204007 with bad export cookie 12734774908413504560 [ 7699.764649] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7699.779457] LustreError: 273875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7711.607340] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7713.945450] Key type lgssc unregistered [ 7714.256594] LNet: 279329:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7714.264162] LNet: Removed LNI 192.168.203.104@tcp [ 7715.068713] Key type .llcrypt unregistered [ 7715.071292] Key type ._llcrypt unregistered [ 7741.211859] alg: No test for adler32 (adler32-zlib) [ 7741.973930] Key type ._llcrypt registered [ 7741.979386] Key type .llcrypt registered [ 7742.101832] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7743.264403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7743.603635] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 7743.817325] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7743.829701] LNet: Accept secure, port 988 [ 7745.479210] Key type lgssc registered [ 7746.498973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7753.725735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.188102] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.241774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7760.361454] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.502783] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7765.475231] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.570581] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.210316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.888383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7776.677827] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7785.392905] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7792.735458] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7801.875030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7802.345328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.434792] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7807.680308] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7816.620185] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7846.368663] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.380554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.396131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.392982] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7847.394065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.400488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.427327] Lustre: Skipped 1 previous similar message [ 7848.825198] Lustre: server umount lustre-OST0000 complete [ 7852.512652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.533613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7857.632853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7857.637540] Lustre: Skipped 2 previous similar messages [ 7858.777948] Lustre: server umount lustre-MDT0000 complete [ 7862.353074] LustreError: 280772:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204169 with bad export cookie 12418047607083983999 [ 7862.353869] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.372388] LustreError: 280772:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7862.715717] Lustre: server umount lustre-MDT0001 complete [ 7873.841862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7876.375551] Key type lgssc unregistered [ 7876.634941] LNet: 285235:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7876.647500] LNet: Removed LNI 192.168.203.104@tcp [ 7877.284655] Key type .llcrypt unregistered [ 7877.293619] Key type ._llcrypt unregistered [ 7891.785914] alg: No test for adler32 (adler32-zlib) [ 7892.540757] Key type ._llcrypt registered [ 7892.546080] Key type .llcrypt registered [ 7892.686387] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7893.816156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7894.267972] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 7894.505496] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7894.509586] LNet: Accept secure, port 988 [ 7896.207257] Key type lgssc registered [ 7897.338064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7904.637667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.130203] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.209246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.366586] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7911.397260] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.528092] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.630845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.948700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.250826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7928.621457] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7934.188359] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7940.389078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7940.723704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7945.528734] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 7953.699662] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7956.149749] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 7968.737680] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7968.749273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.764237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7971.812423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7971.815118] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7971.856493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.344139] Lustre: server umount lustre-OST0000 complete [ 7979.495744] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7979.509637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7979.529059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7979.536341] Lustre: Skipped 1 previous similar message [ 7982.050566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.052652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7982.078828] Lustre: Skipped 1 previous similar message [ 7984.490171] Lustre: server umount lustre-MDT0000 complete [ 7987.169974] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.191460] LustreError: Skipped 1 previous similar message [ 7988.442223] LustreError: 286482:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204295 with bad export cookie 16063821969666043250 [ 7988.446501] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7988.456394] LustreError: 286482:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7988.711885] Lustre: server umount lustre-MDT0001 complete [ 8001.142376] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8004.254661] Key type lgssc unregistered [ 8004.537262] LNet: 290327:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8004.552223] LNet: Removed LNI 192.168.203.104@tcp [ 8005.228367] Key type .llcrypt unregistered [ 8005.232966] Key type ._llcrypt unregistered [ 8026.636424] alg: No test for adler32 (adler32-zlib) [ 8027.397674] Key type ._llcrypt registered [ 8027.399234] Key type .llcrypt registered [ 8027.625555] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8029.082150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8029.798640] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 8029.951209] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8029.953927] LNet: Accept secure, port 988 [ 8031.631575] Key type lgssc registered [ 8033.557693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8044.002279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.701686] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.745580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8050.340763] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8051.176335] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.288512] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.871823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.305256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.171924] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8068.896893] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8073.334241] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8079.241560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8079.533241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.987933] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8090.237697] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8095.919065] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 8104.418996] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8104.428341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.448831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8106.468345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.472563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.590069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.593016] Lustre: Skipped 2 previous similar messages [ 8116.707928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8116.707928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8117.215100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8117.364884] Lustre: server umount lustre-OST0000 complete [ 8120.806238] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8120.814405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.895079] Lustre: server umount lustre-MDT0000 complete [ 8126.945095] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8126.952157] LustreError: Skipped 1 previous similar message [ 8130.304143] LustreError: 291599:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204437 with bad export cookie 6145198954269086510 [ 8130.318181] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8130.331956] LustreError: 291599:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8130.772613] Lustre: server umount lustre-MDT0001 complete [ 8142.778804] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8145.205031] Key type lgssc unregistered [ 8145.466463] LNet: 295298:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8145.474875] LNet: Removed LNI 192.168.203.104@tcp [ 8146.206140] Key type .llcrypt unregistered [ 8146.209600] Key type ._llcrypt unregistered [ 8166.945817] alg: No test for adler32 (adler32-zlib) [ 8167.704534] Key type ._llcrypt registered [ 8167.708195] Key type .llcrypt registered [ 8167.761496] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8168.751054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8169.259791] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 8169.450077] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8169.456438] LNet: Accept secure, port 988 [ 8171.167118] Key type lgssc registered [ 8172.180778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8180.597346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.260674] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.315711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.360546] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.139809] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8192.489276] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8196.581911] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.896691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8198.454246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8203.570530] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8212.878803] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8219.385894] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8225.396804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8225.822218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.204040] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8231.987861] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 8237.003794] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8262.632886] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8262.637163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8262.640510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8262.656815] Lustre: Skipped 1 previous similar message [ 8262.688649] Lustre: Skipped 1 previous similar message [ 8265.162234] Lustre: server umount lustre-OST0000 complete [ 8272.867969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.876146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.891641] Lustre: Skipped 1 previous similar message [ 8272.906390] Lustre: Skipped 1 previous similar message [ 8276.942394] Lustre: server umount lustre-MDT0000 complete [ 8277.986252] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.010289] LustreError: Skipped 1 previous similar message [ 8281.501327] LustreError: 298713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204588 with bad export cookie 1303956343395180566 [ 8281.505158] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8281.518533] LustreError: 298713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8281.889324] Lustre: server umount lustre-MDT0001 complete [ 8293.218828] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8295.924844] Key type lgssc unregistered [ 8296.208059] LNet: 301240:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8296.224567] LNet: Removed LNI 192.168.203.104@tcp [ 8296.936563] Key type .llcrypt unregistered [ 8296.939134] Key type ._llcrypt unregistered [ 8310.800494] alg: No test for adler32 (adler32-zlib) [ 8311.553450] Key type ._llcrypt registered [ 8311.555526] Key type .llcrypt registered [ 8311.705880] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8312.619146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8313.132168] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 8313.316078] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8313.319694] LNet: Accept secure, port 988 [ 8314.967203] Key type lgssc registered [ 8316.080356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8323.705829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.180287] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8325.244258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.594728] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8330.723614] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8335.841358] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.652985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.009637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.555819] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8349.285608] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8355.936852] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.010376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8362.276265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.756235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8371.704927] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8375.608994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 8389.094531] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8389.103634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.125018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8391.139987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8391.142460] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8391.162082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8392.622979] Lustre: server umount lustre-OST0000 complete [ 8399.328055] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8399.338095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.355807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8399.366376] Lustre: Skipped 1 previous similar message [ 8401.376426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.380535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8401.400839] Lustre: Skipped 1 previous similar message [ 8402.698379] Lustre: server umount lustre-MDT0000 complete [ 8406.500554] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.515369] LustreError: Skipped 1 previous similar message [ 8406.803885] LustreError: 302489:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204714 with bad export cookie 12352975119674876778 [ 8406.818264] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8407.276051] Lustre: server umount lustre-MDT0001 complete [ 8418.475395] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8421.484333] Key type lgssc unregistered [ 8421.828266] LNet: 306350:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8421.843624] LNet: Removed LNI 192.168.203.104@tcp [ 8422.741113] Key type .llcrypt unregistered [ 8422.750154] Key type ._llcrypt unregistered [ 8438.089383] alg: No test for adler32 (adler32-zlib) [ 8438.843589] Key type ._llcrypt registered [ 8438.850320] Key type .llcrypt registered [ 8439.005292] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8440.026032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8440.385648] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 8440.572852] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8440.579121] LNet: Accept secure, port 988 [ 8442.279126] Key type lgssc registered [ 8443.107270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8450.451722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.920765] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.976753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.422261] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8457.186865] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.311405] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.076073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.424234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.238798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8473.209672] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8477.738167] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8483.403804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.728651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.888714] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8495.162788] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8502.203370] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 8514.529824] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8514.534909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.549635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8517.604190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8517.606141] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8517.628462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.289405] Lustre: server umount lustre-OST0000 complete [ 8524.769399] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8524.779666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.798859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.802445] Lustre: Skipped 1 previous similar message [ 8527.840621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.843148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.965349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8532.971262] Lustre: Skipped 1 previous similar message [ 8538.081666] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8538.278539] Lustre: server umount lustre-MDT0000 complete [ 8539.105519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8539.120996] LustreError: Skipped 1 previous similar message [ 8542.430135] LustreError: 309569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204849 with bad export cookie 16263371371762200078 [ 8542.432131] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8542.445583] LustreError: 309569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8542.853834] Lustre: server umount lustre-MDT0001 complete [ 8554.190081] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8557.190562] Key type lgssc unregistered [ 8557.500432] LNet: 311298:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8557.516495] LNet: Removed LNI 192.168.203.104@tcp [ 8558.367657] Key type .llcrypt unregistered [ 8558.370917] Key type ._llcrypt unregistered [ 8569.594058] alg: No test for adler32 (adler32-zlib) [ 8570.359410] Key type ._llcrypt registered [ 8570.362404] Key type .llcrypt registered [ 8570.477296] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8586.150626] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8599.409312] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8600.567296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8601.116928] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 8601.381414] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8601.385086] LNet: Accept secure, port 988 [ 8603.063306] Key type lgssc registered [ 8604.132959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8613.154089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8619.932928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8626.212758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8632.189866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8638.263469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8638.345789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.646784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8639.700138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8639.764678] Lustre: lustre-MDT0000: new disk, initializing [ 8639.902642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8639.928882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8644.447412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8655.277482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8655.368612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.432894] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8655.479576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8655.486715] Lustre: Skipped 1 previous similar message [ 8655.590472] Lustre: lustre-MDT0001: new disk, initializing [ 8655.694108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.740504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8655.746366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8659.722279] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8668.840822] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8674.382514] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8681.483517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8681.581127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8681.586628] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8681.851175] Lustre: lustre-OST0000: new disk, initializing [ 8681.860402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8681.984233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8686.783653] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8689.766378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8689.779283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8698.744301] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8703.770094] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8704.010917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8708.807448] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8709.240284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8715.745486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.751726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8715.753329] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8715.764938] Lustre: Skipped 1 previous similar message [ 8715.783386] Lustre: Skipped 1 previous similar message [ 8717.379632] Lustre: server umount lustre-OST0000 complete [ 8720.864663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8720.896162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8720.903253] Lustre: Skipped 1 previous similar message [ 8725.984491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8725.995418] Lustre: Skipped 1 previous similar message [ 8731.104490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.113763] Lustre: Skipped 1 previous similar message [ 8735.203455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8735.336514] Lustre: server umount lustre-MDT0000 complete [ 8737.257402] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.288031] LustreError: Skipped 1 previous similar message [ 8739.420967] LustreError: 315043:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205046 with bad export cookie 17336135241983915587 [ 8739.427954] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8739.763599] Lustre: server umount lustre-MDT0001 complete [ 8748.553416] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:17:34 (1776205054) [ 8755.698633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8756.428532] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.532323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.686692] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8761.825586] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.948824] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.380742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.131983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8776.908751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8785.493199] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8791.940951] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8799.508763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.121196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.621652] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8814.196405] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8824.816155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.819749] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8824.839233] Lustre: Skipped 2 previous similar messages [ 8824.841045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.841054] Lustre: Skipped 2 previous similar messages [ 8829.414329] Lustre: server umount lustre-OST0000 complete [ 8833.505782] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8833.517054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.535420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8833.538229] Lustre: Skipped 1 previous similar message [ 8839.799195] Lustre: server umount lustre-MDT0000 complete [ 8840.161703] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.178288] LustreError: Skipped 1 previous similar message [ 8843.138518] LustreError: 319583:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205150 with bad export cookie 17336135241983916434 [ 8843.143759] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8843.148202] LustreError: 319583:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8854.692218] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8857.349902] Key type lgssc unregistered [ 8857.651801] LNet: 323262:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8857.670136] LNet: Removed LNI 192.168.203.104@tcp [ 8858.479151] Key type .llcrypt unregistered [ 8858.481980] Key type ._llcrypt unregistered [ 8885.831282] alg: No test for adler32 (adler32-zlib) [ 8886.588139] Key type ._llcrypt registered [ 8886.591314] Key type .llcrypt registered [ 8886.738082] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8899.892985] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8901.089384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8901.511441] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 8901.710050] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8901.714534] LNet: Accept secure, port 988 [ 8903.457051] Key type lgssc registered [ 8904.738923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8913.121494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8919.345096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8925.014602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8930.635168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8935.313444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8940.384495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8948.261444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8948.341461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.629853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8949.705490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8949.833268] Lustre: lustre-MDT0000: new disk, initializing [ 8949.981479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.014761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8955.809420] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8966.574061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8966.656408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.771633] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8966.823530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8966.828268] Lustre: Skipped 1 previous similar message [ 8966.948991] Lustre: lustre-MDT0001: new disk, initializing [ 8967.071974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.152062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8967.167101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8972.232445] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 8982.587605] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8989.634431] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8997.983976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8998.074212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8998.081225] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8998.317171] Lustre: lustre-OST2710: new disk, initializing [ 8998.324496] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 8998.430572] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9002.439914] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9004.112575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9004.126643] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9012.827174] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9019.718552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9019.791967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.803596] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9019.945929] Lustre: lustre-OST03e8: new disk, initializing [ 9019.949867] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 9020.036092] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9024.054826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9029.684226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9029.698382] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9031.650115] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9037.754504] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 9037.925917] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9042.222487] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 9042.413966] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9047.254631] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 9047.511510] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9052.035870] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 9052.325802] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9058.703974] Lustre: server umount lustre-MDT0000 complete [ 9059.296192] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9059.307437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.320502] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.837084] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.838469] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.677768] LustreError: 330336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205370 with bad export cookie 6180200568189445139 [ 9062.681163] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.685519] LustreError: 330336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9063.011539] Lustre: server umount lustre-MDT0001 complete [ 9072.607663] Lustre: 332716:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205374/real 1776205374] req@0000000015647d58 x1862486153264576/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776205380 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9072.669234] Lustre: server umount lustre-OST2710 complete [ 9073.122285] Lustre: 325023:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205373/real 1776205373] req@0000000029763283 x1862486153264448/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776205380 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9073.169421] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.182794] Lustre: Skipped 2 previous similar messages [ 9075.169191] Lustre: 325021:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205375/real 1776205375] req@000000008476e5b5 x1862486153264640/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776205382 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9077.189826] Lustre: server umount lustre-OST03e8 complete [ 9095.738858] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 9104.859351] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9114.968154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9121.778518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9128.174711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9134.363996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9141.820562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9141.947053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.308698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9142.372669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9142.489782] Lustre: lustre-MDT0000: new disk, initializing [ 9142.610053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.630698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9147.553812] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9159.020489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9159.076610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.147973] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9159.183887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9159.188237] Lustre: Skipped 1 previous similar message [ 9159.249898] Lustre: lustre-MDT0001: new disk, initializing [ 9159.315748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9159.346041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9159.367028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9163.957416] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9173.786855] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9179.623978] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9185.866665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9185.942464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.950485] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9186.200431] Lustre: lustre-OST0000: new disk, initializing [ 9186.207499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9186.293734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9189.996072] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9191.987493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9191.995902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9198.768389] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9203.306404] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9203.515645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9207.605366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9207.804894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9212.388030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.390416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9212.397381] Lustre: Skipped 1 previous similar message [ 9212.409073] Lustre: Skipped 1 previous similar message [ 9215.527831] Lustre: server umount lustre-OST0000 complete [ 9221.089708] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9221.099818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.112725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9222.624440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9227.747876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9227.752955] Lustre: Skipped 1 previous similar message [ 9233.375434] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9233.477809] LustreError: 339514:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000932a8cdd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9233.527216] Lustre: server umount lustre-MDT0000 complete [ 9233.888894] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.905731] LustreError: Skipped 4 previous similar messages [ 9236.991843] LustreError: 336682:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205544 with bad export cookie 6180200568189446756 [ 9236.995637] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9237.005769] LustreError: 336682:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9245.758755] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:25:51 (1776205551) [ 9247.199199] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9249.172300] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:25:55 (1776205555) [ 9253.249719] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9260.851649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9264.420594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9268.728170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9272.761485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9273.801287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9280.327338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9280.642682] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9280.659541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9280.812925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.567890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9293.849476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.945418] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9293.993671] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9298.148949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9305.846318] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9312.335068] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9319.181945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9319.263896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9319.485864] Lustre: Denying initial registration attempt from nid 192.168.203.104@tcp, specified as failover [ 9319.493492] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9319.514925] LustreError: 343855:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 9319.534719] LustreError: 343855:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9319.543687] LustreError: 343855:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9319.683781] Lustre: server umount lustre-OST0000 complete [ 9319.685911] Lustre: Skipped 1 previous similar message [ 9319.692912] LustreError: 343855:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9326.049124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.067138] Lustre: Skipped 1 previous similar message [ 9326.075451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9326.090409] Lustre: Skipped 4 previous similar messages [ 9335.568824] LustreError: 341892:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205643 with bad export cookie 6180200568189447610 [ 9335.572983] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9335.586786] LustreError: 341892:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9349.182785] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9352.548460] Key type lgssc unregistered [ 9352.806679] LNet: 344930:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9352.814748] LNet: Removed LNI 192.168.203.104@tcp [ 9353.668687] Key type .llcrypt unregistered [ 9353.672563] Key type ._llcrypt unregistered [ 9366.522106] alg: No test for adler32 (adler32-zlib) [ 9367.274495] Key type ._llcrypt registered [ 9367.282926] Key type .llcrypt registered [ 9367.417845] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9368.577308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9368.892574] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 9369.067655] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9369.076560] LNet: Accept secure, port 988 [ 9370.783119] Key type lgssc registered [ 9371.845728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9380.300100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9384.692536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9388.407407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9392.415914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9393.431298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9407.332429] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9416.085103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.416444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9417.448329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9417.618160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.578916] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9429.752788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.876398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9429.900577] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9430.154529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9434.769263] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9442.406623] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9448.198517] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9454.617701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9454.726608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9454.937470] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9455.151510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9460.024987] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9466.997581] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9471.971662] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9471.982472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9472.000134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9477.601227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9477.619544] Lustre: Skipped 2 previous similar messages [ 9482.726869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9482.737601] Lustre: Skipped 1 previous similar message [ 9483.231143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9483.403531] Lustre: server umount lustre-OST0000 complete [ 9487.843425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9487.846794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9487.865036] Lustre: Skipped 2 previous similar messages [ 9487.888938] Lustre: Skipped 2 previous similar messages [ 9492.960888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9492.971679] Lustre: Skipped 1 previous similar message [ 9493.208160] Lustre: server umount lustre-MDT0000 complete [ 9497.078368] LustreError: 347802:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205804 with bad export cookie 9261827576029783962 [ 9497.083110] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9497.093137] LustreError: 347802:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9497.385654] Lustre: server umount lustre-MDT0001 complete [ 9509.216630] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9512.233162] Key type lgssc unregistered [ 9512.499024] LNet: 351507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9512.507155] LNet: Removed LNI 192.168.203.104@tcp [ 9513.290615] Key type .llcrypt unregistered [ 9513.293488] Key type ._llcrypt unregistered [ 9535.508867] alg: No test for adler32 (adler32-zlib) [ 9536.271106] Key type ._llcrypt registered [ 9536.273237] Key type .llcrypt registered [ 9536.382862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9537.333524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9537.681747] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 9537.918861] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9537.924914] LNet: Accept secure, port 988 [ 9539.591213] Key type lgssc registered [ 9540.713406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9548.559725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.178813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.283250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.689516] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9555.424752] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9560.547137] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.585383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.073541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9567.455246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9575.672125] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9581.346608] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9588.757946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9589.129733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9593.599898] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9601.826715] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9608.676341] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 9609.186628] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9609.193822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9609.218162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.745178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.747672] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.777339] Lustre: Skipped 1 previous similar message [ 9613.774607] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 9613.787792] Lustre: Skipped 2 previous similar messages [ 9614.946554] Lustre: server umount lustre-MDT0000 complete [ 9616.865123] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.903163] LustreError: Skipped 2 previous similar messages [ 9619.118517] LustreError: 353758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205926 with bad export cookie 14642001518270495103 [ 9619.119697] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.131248] LustreError: 353758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9621.988120] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9621.989316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.011394] LustreError: Skipped 1 previous similar message [ 9622.041071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9625.495022] Lustre: server umount lustre-MDT0001 complete [ 9632.981450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9641.960411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.710065] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcb32d6cde13e397f to 0xcb32d6cde13e57b6 [ 9652.726954] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 9653.198931] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.222358] LustreError: Skipped 2 previous similar messages [ 9653.374448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.452803] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 9659.462824] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9668.012735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.312770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9673.720105] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 9674.254812] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9675.830125] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 9683.302074] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9685.062365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9691.271811] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9692.929423] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9699.301535] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9699.303464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9699.316106] LustreError: Skipped 1 previous similar message [ 9699.319483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9699.341666] Lustre: Skipped 1 previous similar message [ 9699.356810] Lustre: Skipped 3 previous similar messages [ 9712.095406] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9712.377768] Lustre: server umount lustre-OST0000 complete [ 9719.782748] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9719.790417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.809843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.814233] Lustre: Skipped 6 previous similar messages [ 9723.651059] Lustre: server umount lustre-MDT0000 complete [ 9726.952308] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.976942] LustreError: Skipped 7 previous similar messages [ 9728.209049] LustreError: 352951:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206035 with bad export cookie 14642001518270502838 [ 9728.212939] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9728.222082] LustreError: 352951:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9728.529905] Lustre: server umount lustre-MDT0001 complete [ 9741.543857] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9744.843363] Key type lgssc unregistered [ 9745.195051] LNet: 359476:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9745.213644] LNet: Removed LNI 192.168.203.104@tcp [ 9745.993357] Key type .llcrypt unregistered [ 9746.001683] Key type ._llcrypt unregistered [ 9776.395298] alg: No test for adler32 (adler32-zlib) [ 9777.155449] Key type ._llcrypt registered [ 9777.157935] Key type .llcrypt registered [ 9777.324440] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9779.012790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9779.962553] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 9780.252185] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9780.263499] LNet: Accept secure, port 988 [ 9782.079151] Key type lgssc registered [ 9783.233606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9791.622759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.922416] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.984517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.118230] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.514948] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9803.235171] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.985671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9808.344583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9813.485216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9820.896190] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9826.532355] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9833.958123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9833.961778] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9833.966783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9833.968144] Lustre: Skipped 1 previous similar message [ 9835.855270] Lustre: server umount lustre-MDT0000 complete [ 9838.055098] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9838.091366] LustreError: Skipped 1 previous similar message [ 9839.697504] LustreError: 360942:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206147 with bad export cookie 17583067958334140398 [ 9839.701185] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9839.708844] LustreError: 360942:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9839.994915] Lustre: server umount lustre-MDT0001 complete [ 9854.996377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.330232] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9855.379508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9855.645888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.102095] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9868.251354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.378112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9868.419663] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9868.743920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9873.329569] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9880.601653] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9886.413061] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9894.078399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.275356] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9894.287343] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9894.299640] LustreError: 365653:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 9894.306871] LustreError: 365653:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9894.316118] LustreError: 365653:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9894.457254] Lustre: server umount lustre-OST0000 complete [ 9894.459832] LustreError: 365653:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9901.542375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9901.720464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9901.935512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9906.214644] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9908.346508] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 9913.457404] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9920.554521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9920.661924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.683036] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9920.803328] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9920.946740] Lustre: lustre-OST0001: new disk, initializing [ 9920.953542] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9921.078615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9925.928380] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [ 9930.355360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9930.367280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9934.396051] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9940.973756] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.975648] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9940.976636] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9940.976642] Lustre: Skipped 1 previous similar message [ 9940.994204] Lustre: Skipped 1 previous similar message [ 9941.002531] LustreError: Skipped 1 previous similar message [ 9943.492697] Lustre: server umount lustre-OST0001 complete [ 9946.091844] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.155537] LustreError: Skipped 1 previous similar message [ 9949.154051] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9949.162761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9949.186424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9949.191966] Lustre: Skipped 1 previous similar message [ 9951.202847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9951.216554] Lustre: Skipped 2 previous similar messages [ 9954.825447] Lustre: server umount lustre-OST0000 complete [ 9960.936548] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9960.951933] LustreError: Skipped 1 previous similar message [ 9960.961988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9960.993164] Lustre: Skipped 1 previous similar message [ 9961.007444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.739836] Lustre: server umount lustre-MDT0000 complete [ 9966.561215] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9966.591690] LustreError: Skipped 3 previous similar messages [ 9970.235716] LustreError: 363652:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206277 with bad export cookie 17583067958334140951 [ 9970.242047] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.244623] LustreError: 363652:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9983.652832] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9986.527553] Key type lgssc unregistered [ 9986.792292] LNet: 369023:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9986.805884] LNet: Removed LNI 192.168.203.104@tcp [ 9987.535471] Key type .llcrypt unregistered [ 9987.537390] Key type ._llcrypt unregistered [10019.789629] alg: No test for adler32 (adler32-zlib) [10020.550688] Key type ._llcrypt registered [10020.556210] Key type .llcrypt registered [10020.662885] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10032.618343] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10033.559567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10033.953924] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [10034.120142] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10034.130854] LNet: Accept secure, port 988 [10035.775186] Key type lgssc registered [10036.693843] Lustre: Echo OBD driver; http://www.lustre.org/ [10046.014181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10052.417904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10057.641582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10063.209512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10069.712840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10069.800503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.116115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10071.158586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10071.263438] Lustre: lustre-MDT0000: new disk, initializing [10071.338532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10071.357584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10075.350247] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10087.322826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10087.366547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.438216] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10087.478328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10087.483126] Lustre: Skipped 1 previous similar message [10087.543651] Lustre: lustre-MDT0001: new disk, initializing [10087.650173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10087.687282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10087.697974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10092.259868] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10101.064614] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10105.991888] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10111.957839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10112.013245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10112.020899] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10112.368354] Lustre: lustre-OST0000: new disk, initializing [10112.373677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10112.490227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10117.650373] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10118.736809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10118.762956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10128.645386] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10133.735730] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10134.044594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10138.479860] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10138.714618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10144.736869] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10144.744658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10144.755330] LustreError: Skipped 1 previous similar message [10144.757274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10144.792600] Lustre: Skipped 1 previous similar message [10146.663329] Lustre: server umount lustre-OST0000 complete [10154.464520] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10154.472808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.492634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10154.497300] Lustre: Skipped 1 previous similar message [10160.100825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.120939] Lustre: Skipped 2 previous similar messages [10166.240516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10166.241304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.257800] Lustre: Skipped 3 previous similar messages [10166.424376] Lustre: server umount lustre-MDT0000 complete [10170.186775] LustreError: 373982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206477 with bad export cookie 5020612160981309400 [10170.197914] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10170.208811] LustreError: 373982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10170.630457] Lustre: server umount lustre-MDT0001 complete [10178.961791] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 18:41:25 (1776206485) [10184.381576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10190.000238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10211.362412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10219.758645] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10228.830234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10235.684353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10242.376592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10249.455592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10257.620545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10257.685897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10257.906526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10257.942514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10258.064569] Lustre: lustre-MDT0000: new disk, initializing [10258.212695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10258.243955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10263.228501] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10273.234594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10273.352535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.529929] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10273.590566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10273.598674] Lustre: Skipped 1 previous similar message [10273.749361] Lustre: lustre-MDT0001: new disk, initializing [10273.893445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10273.981174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10273.997677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10280.199620] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10290.593886] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10295.992491] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10301.940543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10302.015227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10302.021711] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10302.289962] Lustre: lustre-OST0000: new disk, initializing [10302.296047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10302.399700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10306.924648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10311.751337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10311.757160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10317.398099] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10323.736536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10323.967964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10329.129125] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10329.358676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10332.640354] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10332.644549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10332.653303] LustreError: Skipped 1 previous similar message [10332.654325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10332.668351] Lustre: Skipped 2 previous similar messages [10332.686716] Lustre: Skipped 1 previous similar message [10337.602781] Lustre: server umount lustre-OST0000 complete [10342.881982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10342.888305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10342.889884] Lustre: Skipped 1 previous similar message [10342.898262] Lustre: Skipped 1 previous similar message [10346.661468] Lustre: server umount lustre-MDT0000 complete [10348.001652] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10348.025496] LustreError: Skipped 1 previous similar message [10349.035105] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.053225] LustreError: Skipped 1 previous similar message [10350.106852] LustreError: 382973:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206657 with bad export cookie 5020612160981310254 [10350.107864] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10350.115955] LustreError: 382973:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [10357.889051] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 18:44:23 (1776206663) [10365.563232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.080168] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.158504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10370.291834] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10371.560314] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10376.678470] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.197337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.446633] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10395.639411] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10404.198290] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10410.849599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10411.211827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10411.218838] Lustre: Skipped 1 previous similar message [10415.933465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10423.064915] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10434.788782] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [10436.671141] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [10439.224367] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [10441.267234] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [10443.178167] Lustre: Failing over lustre-MDT0000 [10443.342881] Lustre: server umount lustre-MDT0000 complete [10443.346129] Lustre: Skipped 1 previous similar message [10444.259416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10444.278477] Lustre: Skipped 2 previous similar messages [10444.280694] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.309114] LustreError: Skipped 2 previous similar messages [10451.423178] Lustre: 371340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206751/real 1776206751] req@000000002cb27bf2 x1862487340276160/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776206758 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10451.454940] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10461.666899] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.696251] LustreError: Skipped 16 previous similar messages [10461.720360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.819073] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x45accc28fa4c1e84 to 0x45accc28fa4c226d [10467.834460] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 0@lo) [10468.123313] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10468.811067] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10472.608067] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10473.470607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10473.496419] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10473.571692] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [10482.156948] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10483.557710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10487.023696] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [10488.771963] Lustre: Failing over lustre-MDT0000 [10488.810397] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.820147] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10488.824450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10488.970420] Lustre: server umount lustre-MDT0000 complete [10493.922408] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.933537] LustreError: Skipped 10 previous similar messages [10501.071370] Lustre: 371339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206801/real 1776206801] req@00000000d6ad51fd x1862487340287104/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776206808 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10501.115830] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.804281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10518.511152] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x45accc28fa4c226d to 0x45accc28fa4c257d [10518.529955] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [10518.534411] Lustre: Skipped 2 previous similar messages [10518.710709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.727372] Lustre: Skipped 1 previous similar message [10518.780772] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10518.991586] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10522.695806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10524.146202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10524.181888] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [10524.224968] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10532.762126] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10534.526766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10544.609083] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10544.612357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.614271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10544.614278] Lustre: Skipped 2 previous similar messages [10544.657905] Lustre: Skipped 4 previous similar messages [10546.454068] Lustre: server umount lustre-MDT0000 complete [10550.607564] LustreError: 385527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206858 with bad export cookie 5020612160981312893 [10550.614591] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10550.620544] LustreError: 385527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10561.503143] Lustre: 391353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206862/real 1776206862] req@00000000e8a9962d x1862487340299200/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206868 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10578.944955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.530629] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.554759] LustreError: Skipped 31 previous similar messages [10584.694634] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10593.697048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10598.196784] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10604.638598] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10609.379955] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10616.482064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10620.666235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10628.477308] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10630.123917] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10640.355224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10640.360908] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10640.374383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10640.388151] Lustre: Skipped 3 previous similar messages [10646.442766] Lustre: server umount lustre-OST0000 complete [10646.445957] Lustre: Skipped 2 previous similar messages [10660.789964] LustreError: 392161:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206968 with bad export cookie 5020612160981313621 [10660.808855] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10660.811207] LustreError: 392161:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10672.750706] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10675.589942] Key type lgssc unregistered [10675.843730] LNet: 395837:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10675.858036] LNet: Removed LNI 192.168.203.104@tcp [10676.707089] Key type .llcrypt unregistered [10676.715594] Key type ._llcrypt unregistered [10697.444566] alg: No test for adler32 (adler32-zlib) [10698.206388] Key type ._llcrypt registered [10698.215842] Key type .llcrypt registered [10698.334395] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10699.295194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10699.773699] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [10699.961804] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10699.965874] LNet: Accept secure, port 988 [10701.607127] Key type lgssc registered [10702.530986] Lustre: Echo OBD driver; http://www.lustre.org/ [10709.374281] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.380816] LustreError: 397326:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10709.387315] LustreError: 397326:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [10709.391381] LustreError: 397326:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10709.399295] LustreError: 397326:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10709.408978] LustreError: 397326:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10715.877408] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.885606] LustreError: 397723:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10715.895846] LustreError: 397723:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [10715.905448] LustreError: 397723:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10715.911842] LustreError: 397723:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10715.920887] LustreError: 397723:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10735.216333] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10739.679812] Key type lgssc unregistered [10740.120806] LNet: 398682:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10740.159822] LNet: Removed LNI 192.168.203.104@tcp [10741.292717] Key type .llcrypt unregistered [10741.295512] Key type ._llcrypt unregistered [10763.257676] alg: No test for adler32 (adler32-zlib) [10764.015273] Key type ._llcrypt registered [10764.021044] Key type .llcrypt registered [10764.159617] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10777.154048] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10778.268132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10778.790633] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [10778.976286] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10778.980755] LNet: Accept secure, port 988 [10780.631267] Key type lgssc registered [10781.856625] Lustre: Echo OBD driver; http://www.lustre.org/ [10790.145332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10797.333194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10804.620229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10812.424585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10822.128737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10822.196736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.405667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10823.461877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10823.579907] Lustre: lustre-MDT0000: new disk, initializing [10823.709104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.761339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10829.222581] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10840.330668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10840.403620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.496298] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10840.534990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10840.541172] Lustre: Skipped 1 previous similar message [10840.686411] Lustre: lustre-MDT0001: new disk, initializing [10840.794617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10840.865357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10840.895865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10846.084374] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10856.470120] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10862.678687] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10869.395591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10869.478431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10869.488415] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10869.770783] Lustre: lustre-OST0000: new disk, initializing [10869.777316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10869.854961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10874.463610] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10876.451854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10876.457845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10883.899328] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10888.987373] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10889.186366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10894.059114] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10894.230681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10896.865932] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10896.866184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.869911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10896.894971] Lustre: Skipped 1 previous similar message [10901.985759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10902.005013] Lustre: Skipped 2 previous similar messages [10902.287937] Lustre: server umount lustre-OST0000 complete [10907.104575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10907.127892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10907.140674] Lustre: Skipped 1 previous similar message [10912.224187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10912.237358] Lustre: Skipped 1 previous similar message [10914.862733] Lustre: server umount lustre-MDT0000 complete [10917.349361] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.365779] LustreError: Skipped 1 previous similar message [10919.270135] LustreError: 401955:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207226 with bad export cookie 9608322660866703653 [10919.279265] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10919.287109] LustreError: 401955:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10919.803562] Lustre: server umount lustre-MDT0001 complete [10929.681826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10931.286929] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:53:57 (1776207237) [10938.224458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.858938] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.933453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10943.969112] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.289222] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10949.089427] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.433893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.740909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10958.088817] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10967.044112] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10974.651352] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10982.114823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10982.701949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10988.557524] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [10996.758448] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11004.456684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11004.539309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11004.549687] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [11004.706340] Lustre: lustre-OST0001: new disk, initializing [11004.713879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [11004.857709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11010.190170] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11014.210968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11014.222686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11019.709165] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11024.873203] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.879886] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11024.888196] Lustre: Skipped 1 previous similar message [11024.910827] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11024.922268] Lustre: Skipped 1 previous similar message [11029.314651] Lustre: server umount lustre-OST0001 complete [11029.988312] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11030.018711] LustreError: Skipped 1 previous similar message [11040.232770] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.255923] LustreError: Skipped 4 previous similar messages [11046.370254] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11046.385362] LustreError: Skipped 1 previous similar message [11046.391975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11046.406025] Lustre: Skipped 1 previous similar message [11046.416014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11046.420160] Lustre: Skipped 1 previous similar message [11050.466602] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11050.485664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.803341] Lustre: server umount lustre-OST0000 complete [11061.844203] Lustre: server umount lustre-MDT0000 complete [11065.458489] LustreError: 406543:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207372 with bad export cookie 9608322660866704507 [11065.460152] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11065.476424] LustreError: 406543:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11084.506111] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11087.389995] Key type lgssc unregistered [11087.730509] LNet: 411469:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11087.742982] LNet: Removed LNI 192.168.203.104@tcp [11088.637769] Key type .llcrypt unregistered [11088.643609] Key type ._llcrypt unregistered [11123.828768] alg: No test for adler32 (adler32-zlib) [11124.583445] Key type ._llcrypt registered [11124.585365] Key type .llcrypt registered [11124.679088] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [11137.383264] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11138.483868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11139.100634] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [11139.349646] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11139.358162] LNet: Accept secure, port 988 [11141.055112] Key type lgssc registered [11142.192461] Lustre: Echo OBD driver; http://www.lustre.org/ [11152.548051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11159.381593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11165.433373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11171.756543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11178.965559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11179.059485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.359785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11180.413047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11180.487090] Lustre: lustre-MDT0000: new disk, initializing [11180.627786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11180.652717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11185.650407] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11196.088968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11196.173700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.301623] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11196.351485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11196.354463] Lustre: Skipped 1 previous similar message [11196.448524] Lustre: lustre-MDT0001: new disk, initializing [11196.538176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11196.590710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11196.601296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11200.764616] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11209.629772] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11214.948530] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11221.421197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11221.510557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11221.522955] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11221.797408] Lustre: lustre-OST0000: new disk, initializing [11221.805551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11221.889838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11226.211452] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11231.814188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11231.826835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11235.039069] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11239.627597] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11239.921665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11244.034639] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11244.225826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11247.584994] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11247.589960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11247.599312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11247.617841] Lustre: Skipped 1 previous similar message [11252.706495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11252.716039] Lustre: Skipped 2 previous similar messages [11257.829979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11257.837748] Lustre: Skipped 1 previous similar message [11259.874879] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11260.045865] Lustre: server umount lustre-OST0000 complete [11268.065914] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11268.068210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11268.069975] LustreError: Skipped 1 previous similar message [11268.081530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11268.096704] Lustre: Skipped 1 previous similar message [11270.772608] Lustre: server umount lustre-MDT0000 complete [11273.185686] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11273.212324] LustreError: Skipped 1 previous similar message [11275.537909] LustreError: 416419:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207583 with bad export cookie 14194812899755801625 [11275.541170] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.554449] LustreError: 416419:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11275.939463] Lustre: server umount lustre-MDT0001 complete [11285.253565] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 18:59:51 (1776207591) [11294.585523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11303.140283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11303.531171] Lustre: lustre-MDT0000: new disk, initializing [11303.571056] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.645179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11307.724307] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11309.025401] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.155316] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11316.234665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11316.587076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11321.400764] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11328.716728] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11334.146842] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11337.190378] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11337.200069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11337.213729] Lustre: Skipped 1 previous similar message [11337.219488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11339.233559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.022211] Lustre: server umount lustre-MDT0000 complete [11344.352404] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.363625] LustreError: Skipped 1 previous similar message [11345.461315] LustreError: 420700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207652 with bad export cookie 14194812899755802479 [11345.477572] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11357.014146] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:01:03 (1776207663) [11364.403067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.962675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.019570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11369.932919] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11378.564583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11383.097991] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11390.234695] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11395.420842] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11401.958185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11402.294242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11402.298858] Lustre: Skipped 1 previous similar message [11407.158589] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11415.041266] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11422.260512] Lustre: Permanently deactivating lustre-OST0000 [11422.276583] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [11432.537361] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11434.335356] LustreError: 426426:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [11434.341311] LustreError: 426426:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11434.349294] LustreError: 426426:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11437.176398] LustreError: 426474:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11437.183215] LustreError: 426474:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11445.095355] Lustre: server umount lustre-OST0000 complete [11445.100535] Lustre: Skipped 1 previous similar message [11447.462389] LustreError: 426725:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11447.471733] LustreError: 426725:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11450.854373] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11450.857155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11450.865909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11450.876025] Lustre: Skipped 1 previous similar message [11450.892055] Lustre: Skipped 3 previous similar messages [11455.455220] Lustre: 426823:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776207756/real 1776207756] req@0000000065d249e7 x1862488498944256/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776207762 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11455.670081] Lustre: server umount lustre-MDT0000 complete [11455.968440] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11455.984603] LustreError: Skipped 4 previous similar messages [11458.933738] LustreError: 426081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207766 with bad export cookie 14194812899755803032 [11458.945345] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11458.946023] LustreError: 426081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [11465.183213] Lustre: 427024:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776207766/real 1776207766] req@0000000020b84c4f x1862488498945344/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776207772 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11473.320703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.530751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11479.757451] LustreError: 427933:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [11485.056294] Lustre: 428280:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.104@tcp [11492.831240] Lustre: 428379:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776207794/real 1776207794] req@000000006dc4d125 x1862488498945856/t0(0) o251->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776207800 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11492.899922] Lustre: server umount MGS complete [11492.905092] Lustre: Skipped 1 previous similar message [11502.990531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.456884] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11503.489117] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11503.497900] Lustre: Skipped 1 previous similar message [11503.558360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11507.285814] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11515.656650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.962307] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11519.904815] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11527.664664] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11533.210873] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11535.316310] Lustre: Permanently reactivating lustre-OST0000 [11535.350271] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [11535.365369] Lustre: Skipped 2 previous similar messages [11564.527235] LustreError: 430943:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11572.625422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11572.989836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11572.999743] Lustre: Skipped 1 previous similar message [11577.980669] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11585.962844] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11597.796699] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11597.808861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.841955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11604.068495] Lustre: server umount lustre-OST0000 complete [11622.367173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11625.956066] LustreError: 429792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207933 with bad export cookie 14194812899755803991 [11625.958314] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11625.965049] LustreError: 429792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11636.034418] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11638.661662] Key type lgssc unregistered [11639.015646] LNet: 433542:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11639.039073] LNet: Removed LNI 192.168.203.104@tcp [11639.918729] Key type .llcrypt unregistered [11639.922681] Key type ._llcrypt unregistered [11660.159969] alg: No test for adler32 (adler32-zlib) [11660.922379] Key type ._llcrypt registered [11660.926597] Key type .llcrypt registered [11661.178103] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [11676.872776] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11677.841509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11678.663737] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [11678.874267] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11678.878397] LNet: Accept secure, port 988 [11680.527115] Key type lgssc registered [11681.578140] Lustre: Echo OBD driver; http://www.lustre.org/ [11690.237703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11696.719312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11703.358580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11709.635728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11716.984027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11717.056810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11718.352987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11718.421883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11718.497207] Lustre: lustre-MDT0000: new disk, initializing [11718.617093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11718.648403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11723.417889] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11734.037727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11734.118320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.201869] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11734.242865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11734.247082] Lustre: Skipped 1 previous similar message [11734.314567] Lustre: lustre-MDT0001: new disk, initializing [11734.424984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.471405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11734.478949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11738.670769] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11747.988584] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11753.670866] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11760.040730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11760.135407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.146600] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11760.481148] Lustre: lustre-OST0000: new disk, initializing [11760.485338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11760.579825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11765.427881] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11767.918055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11767.936000] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11777.271266] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11783.621578] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11783.885949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11789.830274] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11790.178641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11793.891629] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11793.893161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.921628] LustreError: Skipped 1 previous similar message [11793.925874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11793.980631] Lustre: Skipped 1 previous similar message [11798.705373] Lustre: server umount lustre-OST0000 complete [11804.129320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.151662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11804.156833] Lustre: Skipped 2 previous similar messages [11808.736549] Lustre: server umount lustre-MDT0000 complete [11809.249851] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.259037] LustreError: Skipped 1 previous similar message [11810.278431] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11813.284878] LustreError: 436836:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208120 with bad export cookie 13687241660335553267 [11813.285674] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.306361] LustreError: 436836:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11813.642484] Lustre: server umount lustre-MDT0001 complete [11823.506570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11825.502723] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:08:51 (1776208131) [11832.656992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11833.084604] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.092766] LustreError: Skipped 1 previous similar message [11833.155499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11837.139487] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11838.434730] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.553236] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.698789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.038615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11849.057754] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11855.518779] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11860.887752] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11866.774496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11867.020492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11871.177125] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11878.559198] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11880.491512] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11887.404468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11893.732524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11893.745076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11893.746401] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11893.758201] Lustre: Skipped 2 previous similar messages [11893.791170] Lustre: Skipped 1 previous similar message [11898.851177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11898.862295] Lustre: Skipped 1 previous similar message [11899.150068] Lustre: server umount lustre-OST0000 complete [11903.972056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11903.988497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11904.008940] Lustre: Skipped 1 previous similar message [11908.838604] Lustre: server umount lustre-MDT0000 complete [11909.089256] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.113434] LustreError: Skipped 1 previous similar message [11913.379724] LustreError: 443395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208220 with bad export cookie 13687241660335554128 [11913.383573] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11913.397610] LustreError: 443395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11927.762639] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11930.870292] Key type lgssc unregistered [11931.132494] LNet: 445293:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11931.139072] LNet: Removed LNI 192.168.203.104@tcp [11932.111383] Key type .llcrypt unregistered [11932.117975] Key type ._llcrypt unregistered [11959.933892] alg: No test for adler32 (adler32-zlib) [11960.689808] Key type ._llcrypt registered [11960.694695] Key type .llcrypt registered [11960.826750] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11961.215633] Key type .llcrypt unregistered [11961.220175] Key type ._llcrypt unregistered [11976.130338] alg: No test for adler32 (adler32-zlib) [11976.885931] Key type ._llcrypt registered [11976.888324] Key type .llcrypt registered [11977.020601] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11977.924107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11978.378377] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [11978.576850] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11978.591574] LNet: Accept secure, port 988 [11980.247582] Key type lgssc registered [11981.465199] Lustre: Echo OBD driver; http://www.lustre.org/ [11989.275950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.784510] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.820759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11995.282263] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [11996.128575] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.252755] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.354493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12003.703180] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.724257] LustreError: Skipped 1 previous similar message [12003.733465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12005.811533] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [12008.042682] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12010.984913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.007070] LustreError: Skipped 3 previous similar messages [12015.245406] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12021.233131] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12021.252167] LustreError: Skipped 3 previous similar messages [12021.942612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.213925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12026.535667] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12036.576278] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12036.583743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.604805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12037.018703] Lustre: server umount lustre-OST0000 complete [12042.213792] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12042.223636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12042.239888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12044.577995] Lustre: 446937:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208345/real 1776208345] req@00000000bfbc2e44 x1862489379713600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776208352 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12044.616941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.628009] Lustre: Skipped 1 previous similar message [12044.649517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12044.656203] Lustre: Skipped 2 previous similar messages [12046.633906] LustreError: 450085:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000014ad1749) refcount nonzero (2) after lock cleanup; forcing cleanup. [12048.683709] Lustre: server umount lustre-MDT0000 complete [12049.891200] Lustre: 446939:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208350/real 1776208350] req@0000000059ea2cb4 x1862489379714496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776208357 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12049.896929] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12049.918097] Lustre: 446939:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12049.946497] LustreError: Skipped 1 previous similar message [12052.346424] LustreError: 447607:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208359 with bad export cookie 17161213607754438596 [12052.352301] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12052.356604] LustreError: 447607:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12052.780301] Lustre: server umount lustre-MDT0001 complete [12063.842994] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12066.443682] Key type lgssc unregistered [12066.717898] LNet: 450893:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12066.731338] LNet: Removed LNI 192.168.203.104@tcp [12067.474960] Key type .llcrypt unregistered [12067.477393] Key type ._llcrypt unregistered [12088.321918] alg: No test for adler32 (adler32-zlib) [12089.080574] Key type ._llcrypt registered [12089.082397] Key type .llcrypt registered [12089.192608] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12090.044959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12090.483634] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [12090.757912] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12090.765544] LNet: Accept secure, port 988 [12092.431342] Key type lgssc registered [12093.555129] Lustre: Echo OBD driver; http://www.lustre.org/ [12101.320316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.945090] LustreError: 452330:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 1: rc = -5 [12149.663405] LustreError: 452330:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [12179.423352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12183.863966] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12190.857473] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12197.443652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12202.255974] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12202.376770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12202.447258] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [12207.078968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12207.594231] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.737710] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12215.604577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.962966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12220.796862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12228.174048] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12234.751186] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12243.424893] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12243.432502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12243.452284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12244.961189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12244.963592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.935895] Lustre: server umount lustre-OST0000 complete [12255.200732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.204141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.232105] Lustre: Skipped 1 previous similar message [12255.269395] Lustre: Skipped 2 previous similar messages [12259.194485] Lustre: server umount lustre-MDT0000 complete [12260.322334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.338964] LustreError: Skipped 1 previous similar message [12263.335874] LustreError: 452337:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208570 with bad export cookie 9869592965124213137 [12263.347648] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12263.352369] LustreError: 452337:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12263.781597] Lustre: server umount lustre-MDT0001 complete [12277.253509] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12280.410741] Key type lgssc unregistered [12280.759118] LNet: 456026:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12280.769403] LNet: Removed LNI 192.168.203.104@tcp [12281.620846] Key type .llcrypt unregistered [12281.628776] Key type ._llcrypt unregistered [12308.001960] alg: No test for adler32 (adler32-zlib) [12308.775809] Key type ._llcrypt registered [12308.787438] Key type .llcrypt registered [12308.929204] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12310.177220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12311.506963] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [12311.899541] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12311.905754] LNet: Accept secure, port 988 [12313.623111] Key type lgssc registered [12314.780930] Lustre: Echo OBD driver; http://www.lustre.org/ [12323.779906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12325.678529] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12325.756080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12330.979981] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12331.018380] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12336.100231] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12341.098175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12341.217462] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12341.526557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12346.130914] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12356.464961] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12363.381835] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12371.115463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12371.534592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12375.619954] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [12376.554828] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12385.156717] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12390.886159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12390.908032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12390.927710] Lustre: Skipped 2 previous similar messages [12390.931837] Lustre: Skipped 1 previous similar message [12391.914203] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [12391.918093] Lustre: Skipped 1 previous similar message [12395.419250] Lustre: server umount lustre-MDT0000 complete [12396.001794] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12396.017341] LustreError: Skipped 2 previous similar messages [12398.253231] LustreError: 457492:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.4@tcp arrived at 1776208705 with bad export cookie 5871589969611702085 [12400.529542] LustreError: 457492:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208708 with bad export cookie 5871589969611701189 [12400.542633] LustreError: 457492:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [12400.557464] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12401.136095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12401.163980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12405.152975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12405.153127] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.168733] LustreError: Skipped 4 previous similar messages [12406.783391] Lustre: server umount lustre-OST0000 complete [12413.878489] Lustre: server umount lustre-MDT0001 complete [12425.854118] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12428.914420] Key type lgssc unregistered [12429.227673] LNet: 461320:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12429.276452] LNet: Removed LNI 192.168.203.104@tcp [12430.091463] Key type .llcrypt unregistered [12430.096433] Key type ._llcrypt unregistered [12455.003525] alg: No test for adler32 (adler32-zlib) [12455.757419] Key type ._llcrypt registered [12455.759426] Key type .llcrypt registered [12455.873748] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12456.624351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12457.354815] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [12457.545629] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12457.548499] LNet: Accept secure, port 988 [12459.312262] Key type lgssc registered [12460.618430] Lustre: Echo OBD driver; http://www.lustre.org/ [12468.810637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12470.527513] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.614957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12475.277140] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12475.875625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12481.005697] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12483.905594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12484.253994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12488.820160] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12495.171927] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12500.198142] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12506.148856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12506.556891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12510.647624] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12518.035825] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12520.898671] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:193 [12526.052230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12526.079130] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12526.090644] Lustre: Skipped 1 previous similar message [12529.125493] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [12531.185202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12531.192650] Lustre: Skipped 1 previous similar message [12534.232838] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [12539.353767] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [12539.357192] Lustre: Skipped 2 previous similar messages [12540.385579] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [12540.577809] Lustre: server umount lustre-MDT0001 complete [12541.408466] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12541.428821] LustreError: Skipped 1 previous similar message [12546.528701] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12546.550090] LustreError: Skipped 1 previous similar message [12551.649563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12551.677488] Lustre: Skipped 1 previous similar message [12551.695091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12553.819352] Lustre: server umount lustre-OST0000 complete [12556.770508] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.792309] LustreError: Skipped 1 previous similar message [12564.447161] Lustre: 465835:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208865/real 1776208865] req@0000000083bfcf8e x1862489881985536/t0(0) o251->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776208871 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12564.671891] Lustre: server umount lustre-MDT0000 complete [12580.722405] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12584.147783] Key type lgssc unregistered [12584.519982] LNet: 466591:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12584.529915] LNet: Removed LNI 192.168.203.104@tcp [12585.729601] Key type .llcrypt unregistered [12585.745197] Key type ._llcrypt unregistered [12613.718204] alg: No test for adler32 (adler32-zlib) [12614.474152] Key type ._llcrypt registered [12614.479218] Key type .llcrypt registered [12614.612281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [12632.621318] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12634.497584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12635.495070] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [12635.768570] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12635.772287] LNet: Accept secure, port 988 [12637.407117] Key type lgssc registered [12639.049273] Lustre: Echo OBD driver; http://www.lustre.org/ [12649.399534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12656.255091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12662.406705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12668.867448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12677.297837] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12686.172130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12692.793274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12692.855975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.151900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12694.186077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12694.274564] Lustre: lustre-MDT0000: new disk, initializing [12694.352573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12694.380179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12699.251411] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12707.703364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12713.681432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12713.747568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12713.753543] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12713.972249] Lustre: lustre-OST0000: new disk, initializing [12713.982276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12713.991848] Lustre: Skipped 1 previous similar message [12714.060822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12718.613106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12727.278593] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12734.917432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12741.269585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12741.385607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12741.479631] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12741.524309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12741.650963] Lustre: lustre-MDT0001: new disk, initializing [12741.715636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12741.747796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12741.792198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12746.039721] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12755.939765] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [12756.121293] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12760.929841] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [12761.170827] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12768.740102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12768.744263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12768.745433] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12768.748482] Lustre: Skipped 1 previous similar message [12768.772161] Lustre: Skipped 1 previous similar message [12770.284986] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [12771.362850] Lustre: server umount lustre-MDT0001 complete [12773.864169] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12773.872989] LustreError: Skipped 1 previous similar message [12775.376048] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.977414] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12779.001825] LustreError: Skipped 1 previous similar message [12784.108849] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.125587] LustreError: Skipped 1 previous similar message [12789.216589] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12789.218440] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.227136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.229172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12794.373732] Lustre: server umount lustre-OST0000 complete [12809.695170] Lustre: 475028:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776209111/real 1776209111] req@00000000a6f6cf65 x1862490068634432/t0(0) o251->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776209117 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12809.854668] Lustre: server umount lustre-MDT0000 complete [12822.530973] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12824.961849] Key type lgssc unregistered [12825.210425] LNet: 475784:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12825.217117] LNet: Removed LNI 192.168.203.104@tcp [12825.793722] Key type .llcrypt unregistered [12825.796329] Key type ._llcrypt unregistered [12875.323717] alg: No test for adler32 (adler32-zlib) [12876.084710] Key type ._llcrypt registered [12876.099745] Key type .llcrypt registered [12876.225133] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12876.592557] Key type .llcrypt unregistered [12876.594375] Key type ._llcrypt unregistered [12892.916394] alg: No test for adler32 (adler32-zlib) [12893.678852] Key type ._llcrypt registered [12893.689681] Key type .llcrypt registered [12893.802382] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12895.146723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12895.721808] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [12895.916246] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12895.919762] LNet: Accept secure, port 988 [12897.583143] Key type lgssc registered [12898.662549] Lustre: Echo OBD driver; http://www.lustre.org/ [12907.121900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12913.749900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12919.362471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12926.580470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12926.643368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12927.887386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12927.923240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12928.046790] Lustre: lustre-MDT0000: new disk, initializing [12928.232535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12928.274368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12933.113320] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12943.659558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12943.731575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12943.856428] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12943.922440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12943.934346] Lustre: Skipped 1 previous similar message [12944.022993] Lustre: lustre-MDT0001: new disk, initializing [12944.105940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12944.144543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12944.153192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12948.379379] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12957.176648] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12962.672905] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12968.908805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12968.981933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12968.989848] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12969.353970] Lustre: lustre-OST0000: new disk, initializing [12969.362956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12969.455180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12973.745551] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [12974.702051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12974.717496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12983.100870] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12989.402193] Lustre: server umount lustre-MDT0000 complete [12990.445414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.448545] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12990.455690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.460130] Lustre: Skipped 1 previous similar message [12992.890505] LustreError: 479305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209300 with bad export cookie 13786288823358910308 [12992.906131] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12992.907856] LustreError: 479305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [12995.552171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12995.556633] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12995.560701] Lustre: Skipped 1 previous similar message [12995.561558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12995.588711] LustreError: Skipped 2 previous similar messages [12997.612197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12997.615448] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.626042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12999.280443] Lustre: server umount lustre-MDT0001 complete [13002.868520] Lustre: server umount lustre-OST0000 complete [13013.142360] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 19:28:39 (1776209319) [13014.140601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13019.987732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13020.404635] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.429171] LustreError: Skipped 1 previous similar message [13020.492591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13025.192890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [13025.764297] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13033.654546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13034.161103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13038.251879] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [13044.842411] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13050.024889] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13056.382589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13056.459518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13056.621704] Lustre: Found index 0 for lustre-OST0000, updating log [13056.797447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13062.000766] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [13070.051862] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13071.153477] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [13081.578244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.581610] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13081.584106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13081.594157] Lustre: Skipped 1 previous similar message [13086.657570] Lustre: server umount lustre-OST0000 complete [13090.784755] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13090.792539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13090.810875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13090.813704] Lustre: Skipped 1 previous similar message [13097.840371] Lustre: server umount lustre-MDT0000 complete [13101.454768] LustreError: 485342:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209408 with bad export cookie 13786288823358911652 [13101.457310] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13101.480014] LustreError: 485342:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [13109.881498] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 19:30:15 (1776209415) [13115.293590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13119.859676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13124.904057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13129.095744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13145.858849] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13156.468239] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13170.317994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13178.198574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13183.582221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13189.263173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13195.730944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13195.799590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13196.072085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13196.159193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [13196.302752] Lustre: lustre-MDT0000: new disk, initializing [13196.491269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13196.529757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13202.100249] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [13214.339051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13214.461476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13214.572891] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13214.664186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [13214.672409] Lustre: Skipped 1 previous similar message [13214.806347] Lustre: lustre-MDT0001: new disk, initializing [13215.014205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13215.037903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13220.329956] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [13230.423628] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13235.855479] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13242.122667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13242.186363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13242.194301] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [13242.401334] Lustre: lustre-OST0000: new disk, initializing [13242.405315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [13242.495555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13242.502292] Lustre: Skipped 1 previous similar message [13246.451641] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all 8 [13250.621529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13250.633381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13255.283941] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13259.467266] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13259.675377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13263.728060] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [13263.940870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13266.405340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13266.407452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13266.409335] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13266.420707] Lustre: Skipped 2 previous similar messages [13266.441872] Lustre: Skipped 7 previous similar messages [13271.780284] Lustre: server umount lustre-OST0000 complete [13271.785264] Lustre: Skipped 1 previous similar message [13276.641571] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13276.646590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13276.649747] LustreError: Skipped 1 previous similar message [13276.669309] Lustre: Skipped 1 previous similar message [13281.762044] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13281.778690] LustreError: Skipped 2 previous similar messages [13284.728982] LustreError: 490414:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209592 with bad export cookie 13786288823358912723 [13284.731705] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13284.736361] LustreError: 490414:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [13305.367872] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13311.745392] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13320.052323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13326.415319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13331.571780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13337.166152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13338.561361] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13052 sec ========= 19:34:04 (1776209644) [13374.594856] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13377.163866] Key type lgssc unregistered [13377.416489] LNet: 499154:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13377.428863] LNet: Removed LNI 192.168.203.104@tcp [13378.143282] Key type .llcrypt unregistered [13378.148263] Key type ._llcrypt unregistered