[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454001638 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002226] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006464] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.010059] LSM: Security Framework initializing [ 0.011021] Yama: becoming mindful. [ 0.011800] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.019324] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023571] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027024] *** VALIDATE tmpfs *** [ 0.028304] *** VALIDATE proc *** [ 0.029000] *** VALIDATE cgroup *** [ 0.029011] *** VALIDATE cgroup2 *** [ 0.030249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033038] Spectre V2 : User space: Vulnerable [ 0.034009] Speculative Store Bypass: Vulnerable [ 0.037455] debug: unmapping init [mem 0xffffffff9d459000-0xffffffff9d460fff] [ 0.039165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040740] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041021] ... version: 2 [ 0.042011] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.044009] ... value mask: 0000ffffffffffff [ 0.045024] ... max period: 00007fffffffffff [ 0.046012] ... fixed-purpose events: 3 [ 0.047010] ... event mask: 000000070000000f [ 0.048281] rcu: Hierarchical SRCU implementation. [ 0.050428] smp: Bringing up secondary CPUs ... [ 0.051554] x86: Booting SMP configuration: [ 0.052019] .... node #0, CPUs: #1 #2 #3 [ 0.055498] smp: Brought up 1 node, 4 CPUs [ 0.057013] smpboot: Max logical packages: 1 [ 0.058042] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143020] node 0 deferred pages initialised in 83ms [ 0.147181] devtmpfs: initialized [ 0.148272] x86/mm: Memory block size: 128MB [ 0.150646] gcov: version magic: 0x41383552 [ 0.152170] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154260] pinctrl core: initialized pinctrl subsystem [ 0.155196] [ 0.155672] ************************************************************* [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157009] ** ** [ 0.158011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159010] ** ** [ 0.160013] ** This means that this kernel is built to expose internal ** [ 0.161010] ** IOMMU data structures, which may compromise security on ** [ 0.162009] ** your system. ** [ 0.163011] ** ** [ 0.164012] ** If you see this message and you are not debugging the ** [ 0.165012] ** kernel, report this immediately to your vendor! ** [ 0.166011] ** ** [ 0.167009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168014] ************************************************************* [ 0.169551] NET: Registered protocol family 16 [ 0.170331] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173364] cpuidle: using governor menu [ 0.174566] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176285] PCI: Using configuration type 1 for base access [ 0.177127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187150] cryptd: max_cpu_qlen set to 1000 [ 0.189203] ACPI: Added _OSI(Module Device) [ 0.190009] ACPI: Added _OSI(Processor Device) [ 0.191007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196139] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198459] ACPI: Interpreter enabled [ 0.199039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200011] ACPI: Using IOAPIC for interrupt routing [ 0.201089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202351] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.210493] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.211020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.213163] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218424] acpiphp: Slot [2] registered [ 0.219083] acpiphp: Slot [5] registered [ 0.220114] acpiphp: Slot [6] registered [ 0.222082] acpiphp: Slot [7] registered [ 0.223071] acpiphp: Slot [8] registered [ 0.224066] acpiphp: Slot [9] registered [ 0.225065] acpiphp: Slot [10] registered [ 0.226113] acpiphp: Slot [3] registered [ 0.227061] acpiphp: Slot [4] registered [ 0.229084] acpiphp: Slot [11] registered [ 0.230073] acpiphp: Slot [12] registered [ 0.231071] acpiphp: Slot [13] registered [ 0.232064] acpiphp: Slot [14] registered [ 0.234076] acpiphp: Slot [15] registered [ 0.235063] acpiphp: Slot [16] registered [ 0.236088] acpiphp: Slot [17] registered [ 0.237060] acpiphp: Slot [18] registered [ 0.238066] acpiphp: Slot [19] registered [ 0.240070] acpiphp: Slot [20] registered [ 0.241065] acpiphp: Slot [21] registered [ 0.242062] acpiphp: Slot [22] registered [ 0.243065] acpiphp: Slot [23] registered [ 0.244075] acpiphp: Slot [24] registered [ 0.245059] acpiphp: Slot [25] registered [ 0.246069] acpiphp: Slot [26] registered [ 0.247074] acpiphp: Slot [27] registered [ 0.248059] acpiphp: Slot [28] registered [ 0.250102] acpiphp: Slot [29] registered [ 0.251000] acpiphp: Slot [30] registered [ 0.251000] acpiphp: Slot [31] registered [ 0.253055] PCI host bridge to bus 0000:00 [ 0.255015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.257020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.259017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.262017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.267025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.269176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.271974] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275436] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.285013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.289056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.292015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.294014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.296017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.299515] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301740] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.304028] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.306657] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.310820] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.320871] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.326013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332178] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.337019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.344022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.358011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.367371] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.376029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.383015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.398017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.410540] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.417020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.423015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.437018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.446938] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.455013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.464021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.481014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.496779] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.502017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.511018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.536015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.544984] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.553014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.559014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.574015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.584480] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.587354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.590335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.592331] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.594183] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.598101] iommu: Default domain type: Passthrough [ 0.600469] SCSI subsystem initialized [ 0.602099] ACPI: bus type USB registered [ 0.603089] usbcore: registered new interface driver usbfs [ 0.605057] usbcore: registered new interface driver hub [ 0.607074] usbcore: registered new device driver usb [ 0.608000] pps_core: LinuxPPS API ver. 1 registered [ 0.608000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.612083] PTP clock support registered [ 0.614073] EDAC MC: Ver: 3.0.0 [ 0.615373] PCI: Using ACPI for IRQ routing [ 0.616492] NetLabel: Initializing [ 0.617010] NetLabel: domain hash size = 128 [ 0.619008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.620053] NetLabel: unlabeled traffic allowed by default [ 0.622043] vgaarb: loaded [ 0.623221] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.624011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.629287] clocksource: Switched to clocksource kvm-clock [ 0.721411] VFS: Disk quotas dquot_6.6.0 [ 0.722860] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.724743] *** VALIDATE ramfs *** [ 0.725757] *** VALIDATE hugetlbfs *** [ 0.727242] pnp: PnP ACPI init [ 0.729311] pnp: PnP ACPI: found 6 devices [ 0.744326] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.747552] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.749683] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.751790] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.753421] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.755153] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.757483] NET: Registered protocol family 2 [ 0.759486] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.763628] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.766605] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.771285] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.774384] TCP: Hash tables configured (established 65536 bind 65536) [ 0.776736] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.779310] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.781693] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.784234] NET: Registered protocol family 1 [ 0.786547] RPC: Registered named UNIX socket transport module. [ 0.788309] RPC: Registered udp transport module. [ 0.789671] RPC: Registered tcp transport module. [ 0.790972] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.792775] NET: Registered protocol family 44 [ 0.794073] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.795774] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.797515] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.799466] PCI: CLS 0 bytes, default 64 [ 0.800908] Unpacking initramfs... [ 2.112584] debug: unmapping init [mem 0xffff8bab3cc54000-0xffff8bab3ffbffff] [ 2.115143] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.116771] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.119514] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.610186] Initialise system trusted keyrings [ 2.611225] Key type blacklist registered [ 2.612756] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.622105] zbud: loaded [ 2.624339] *** VALIDATE nfs *** [ 2.625474] *** VALIDATE nfs4 *** [ 2.628416] pstore: using deflate compression [ 2.632132] Platform Keyring initialized [ 2.725355] NET: Registered protocol family 38 [ 2.727056] Key type asymmetric registered [ 2.728553] Asymmetric key parser 'x509' registered [ 2.730456] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.733893] io scheduler mq-deadline registered [ 2.735568] io scheduler kyber registered [ 2.737260] io scheduler bfq registered [ 2.738710] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.741398] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.743886] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.746647] ACPI: Power Button [PWRF] [ 2.753282] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.760750] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.775721] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.785518] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.808406] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.835084] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.863848] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.868569] Non-volatile memory driver v1.3 [ 2.870144] Linux agpgart interface v0.103 [ 2.900744] virtio_blk virtio1: [vda] 68264 512-byte logical blocks (35.0 MB/33.3 MiB) [ 2.903723] vda: detected capacity change from 0 to 34951168 [ 2.926239] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.928722] vdb: detected capacity change from 0 to 1073741824 [ 2.942421] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.945030] vdc: detected capacity change from 0 to 2621440000 [ 2.957530] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.960421] vdd: detected capacity change from 0 to 2621440000 [ 2.973497] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.976421] vde: detected capacity change from 0 to 4294967296 [ 2.990156] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.992832] vdf: detected capacity change from 0 to 4294967296 [ 2.999684] libphy: Fixed MDIO Bus: probed [ 3.009811] usbcore: registered new interface driver usbserial_generic [ 3.011636] usbserial: USB Serial support registered for generic [ 3.013201] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.016244] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.017709] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.019495] mousedev: PS/2 mouse device common for all mice [ 3.022833] rtc_cmos 00:05: RTC can wake from S4 [ 3.024806] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.029280] rtc_cmos 00:05: registered as rtc0 [ 3.029422] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.031188] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.036112] intel_pstate: CPU model not supported [ 3.037925] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.039840] hid: raw HID events driver (C) Jiri Kosina [ 3.042936] usbcore: registered new interface driver usbhid [ 3.044177] usbhid: USB HID core driver [ 3.045102] drop_monitor: Initializing network drop monitor service [ 3.046475] Initializing XFRM netlink socket [ 3.047649] NET: Registered protocol family 10 [ 3.049418] Segment Routing with IPv6 [ 3.050252] NET: Registered protocol family 17 [ 3.051285] mpls_gso: MPLS GSO support [ 3.055474] RAS: Correctable Errors collector initialized. [ 3.057471] AVX version of gcm_enc/dec engaged. [ 3.059176] AES CTR mode by8 optimization enabled [ 3.118617] sched_clock: Marking stable (3118596231, 0)->(3955215492, -836619261) [ 3.120914] registered taskstats version 1 [ 3.121967] Loading compiled-in X.509 certificates [ 3.123460] zswap: loaded using pool lzo/zbud [ 3.142694] Key type big_key registered [ 3.153668] Key type encrypted registered [ 3.155368] ima: No TPM chip found, activating TPM-bypass! [ 3.157488] ima: Allocated hash algorithm: sha1 [ 3.159289] ima: No architecture policies found [ 3.160873] evm: Initialising EVM extended attributes: [ 3.162696] evm: security.selinux [ 3.163879] evm: security.ima [ 3.164895] evm: security.capability [ 3.166174] evm: HMAC attrs: 0x1 [ 3.169198] rtc_cmos 00:05: setting system clock to 2026-06-09 23:10:19 UTC (1781046619) [ 3.174961] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 3.177811] debug: unmapping init [mem 0xffffffff9d182000-0xffffffff9d458fff] [ 3.190163] Write protecting the kernel read-only data: 28672k [ 3.194124] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.196698] debug: unmapping init [mem 0xffffffff9c114000-0xffffffff9c1fffff] [ 3.230025] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.237677] systemd[1]: Detected virtualization kvm. [ 3.239471] systemd[1]: Detected architecture x86-64. [ 3.241338] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.264684] systemd[1]: No hostname configured. [ 3.266096] systemd[1]: Set hostname to . [ 3.267629] random: systemd: uninitialized urandom read (16 bytes read) [ 3.270091] systemd[1]: Initializing machine ID from random generator. [ 3.316319] random: ln: uninitialized urandom read (6 bytes read) [ 3.392034] random: systemd: uninitialized urandom read (16 bytes read) [ 3.394339] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.399826] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.403801] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.913738] device-mapper: uevent: version 1.0.3 [ 3.915098] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.240744] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.549430] virtio_net virtio0 ens2: renamed from eth0 [ 4.613866] scsi host0: ata_piix [ 4.659328] scsi host1: ata_piix [ 4.660859] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.663100] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.833526] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.672144] random: crng init done [ 9.673598] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.881455] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.882242] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.087758] SELinux: Disabled at runtime. [ 11.145941] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.153703] systemd[1]: Detected virtualization kvm. [ 11.155708] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.596408] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.600102] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.604283] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.609256] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.613814] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.620381] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.625526] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ 11.672707] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.978405] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.298403] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.345472] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.425095] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.441352] EDAC sbridge: Ver: 1.1.2 [ 13.677071] Key type dns_resolver registered [ 13.973394] NFS: Registering the id_resolver key type [ 13.975048] Key type id_resolver registered [ 13.976263] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg131-server login: [ 39.440716] libcfs: loading out-of-tree module taints kernel. [ 39.472072] alg: No test for adler32 (adler32-zlib) [ 40.222309] Key type ._llcrypt registered [ 40.223069] Key type .llcrypt registered [ 40.277834] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 51.004697] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 51.861431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.219642] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 52.671996] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 52.674622] LNet: Accept secure, port 988 [ 54.353110] Key type lgssc registered [ 55.152879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 63.723262] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.028625] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.534768] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.876124] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.575882] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.907629] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 91.406336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.457098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.564317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 92.589229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 92.670583] Lustre: lustre-MDT0000: new disk, initializing [ 92.719956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.725948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 95.144849] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 101.910591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.953583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 101.987049] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 101.998421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 102.001464] Lustre: Skipped 1 previous similar message [ 102.046976] Lustre: lustre-MDT0001: new disk, initializing [ 102.079813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.102072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 102.113159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.991082] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 112.499879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.550979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 112.556057] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 112.699579] Lustre: lustre-OST0000: new disk, initializing [ 112.703451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 112.735096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 114.976535] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 117.254692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 117.263275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 122.091081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.123171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 122.127334] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 122.175079] Lustre: lustre-OST0001: new disk, initializing [ 122.177493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 122.199277] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 122.805741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 122.811653] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 124.203892] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 131.170322] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 135.778670] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 143.281165] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.597845] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing yml_node [ 147.862502] Lustre: DEBUG MARKER: Client: 2.15.8.5 [ 148.994223] Lustre: DEBUG MARKER: MDS: 2.15.8.5 [ 150.199542] Lustre: DEBUG MARKER: OSS: 2.15.8.5 [ 150.998626] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 19:12:47 EDT 2026 [ 154.949913] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 155.684399] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 168.928734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.928919] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 168.929155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.934312] Lustre: Skipped 3 previous similar messages [ 173.848839] Lustre: server umount lustre-MDT0000 complete [ 174.053076] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 174.058345] LustreError: Skipped 3 previous similar messages [ 175.492161] LustreError: 6227:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046791 with bad export cookie 3524707905779799259 [ 175.493720] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 175.497348] LustreError: 6227:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 179.168746] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 179.168781] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 179.169139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 179.169144] Lustre: Skipped 3 previous similar messages [ 179.173870] LustreError: Skipped 1 previous similar message [ 179.178265] Lustre: Skipped 1 previous similar message [ 180.192841] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 180.192939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 180.198640] LustreError: Skipped 1 previous similar message [ 180.204778] Lustre: Skipped 2 previous similar messages [ 181.735880] Lustre: server umount lustre-MDT0001 complete [ 183.452437] Lustre: server umount lustre-OST0000 complete [ 188.566525] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 192.468464] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 196.988654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.964648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.442541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.102754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.129860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.159889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.268878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 208.282696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 208.335977] Lustre: lustre-MDT0000: new disk, initializing [ 208.382629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.395355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.394762] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 215.677441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.709620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.749370] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 215.869822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.691763] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 222.410322] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 225.027512] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 227.997432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.027735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.031452] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 228.159691] Lustre: lustre-OST0000: new disk, initializing [ 228.164982] Lustre: Skipped 1 previous similar message [ 228.168903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 228.172064] Lustre: Skipped 2 previous similar messages [ 228.204888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.207834] Lustre: Skipped 1 previous similar message [ 229.566407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.569583] Lustre: Skipped 1 previous similar message [ 229.571615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.028143] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 234.493159] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 236.211972] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 236.296139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 237.909215] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 237.994451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 240.097938] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 240.101166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.105334] Lustre: Skipped 1 previous similar message [ 240.106801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 240.113289] Lustre: Skipped 3 previous similar messages [ 244.926171] Lustre: server umount lustre-OST0000 complete [ 244.930505] Lustre: Skipped 1 previous similar message [ 246.753818] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 246.759707] LustreError: Skipped 1 previous similar message [ 246.761871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.767522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 252.808933] Lustre: server umount lustre-MDT0000 complete [ 254.697115] LustreError: 14308:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046871 with bad export cookie 3524707905779800715 [ 254.703114] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 254.712061] LustreError: 14308:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 256.909888] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:14:33 (1781046873) [ 258.051676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 258.951677] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:14:35 (1781046875) [ 262.091076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.304727] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.312223] LustreError: Skipped 2 previous similar messages [ 262.334613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.126793] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 267.684793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.745194] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.742177] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 273.307284] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 276.188564] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 279.174789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 281.305032] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 284.713806] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 298.977000] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 298.979793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.984270] Lustre: Skipped 1 previous similar message [ 298.986473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 298.989292] Lustre: Skipped 2 previous similar messages [ 303.075117] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 304.536819] Lustre: server umount lustre-OST0000 complete [ 304.539130] Lustre: Skipped 1 previous similar message [ 308.193256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.198478] Lustre: Skipped 2 previous similar messages [ 313.314421] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.326030] LustreError: Skipped 1 previous similar message [ 314.000184] LustreError: 18840:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046930 with bad export cookie 3524707905779801569 [ 314.001476] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.005259] LustreError: 18840:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 320.519293] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 321.904188] Key type lgssc unregistered [ 322.039474] LNet: 22531:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.043608] LNet: Removed LNI 192.168.201.131@tcp [ 322.438414] Key type .llcrypt unregistered [ 322.439553] Key type ._llcrypt unregistered [ 333.794876] alg: No test for adler32 (adler32-zlib) [ 334.546307] Key type ._llcrypt registered [ 334.547295] Key type .llcrypt registered [ 334.604241] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 335.147614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 335.311059] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 335.414310] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 335.416287] LNet: Accept secure, port 988 [ 337.016108] Key type lgssc registered [ 337.508472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 340.756654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.948192] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.977969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.779844] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 347.105191] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.335940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.480934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 349.151853] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 352.172462] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 354.655252] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 357.385322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.530188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.217835] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 362.696049] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 371.878446] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 379.360939] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 379.364198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.370163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 379.729644] Lustre: server umount lustre-OST0000 complete [ 382.433188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.437166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 382.439768] Lustre: Skipped 1 previous similar message [ 387.040174] Lustre: 26863:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781046997/real 1781046997] req@000000001b7cd2e0 x1867563087703424/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781047003 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 387.161495] Lustre: server umount lustre-MDT0000 complete [ 387.555774] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.570387] LustreError: Skipped 1 previous similar message [ 388.860400] LustreError: 23975:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047005 with bad export cookie 1982537977141887192 [ 388.861239] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.865492] LustreError: 23975:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 389.006624] Lustre: server umount lustre-MDT0001 complete [ 395.149619] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 396.339576] Key type lgssc unregistered [ 396.475850] LNet: 27667:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.479578] LNet: Removed LNI 192.168.201.131@tcp [ 396.808912] Key type .llcrypt unregistered [ 396.810377] Key type ._llcrypt unregistered [ 407.951775] alg: No test for adler32 (adler32-zlib) [ 408.703491] Key type ._llcrypt registered [ 408.705124] Key type .llcrypt registered [ 408.764994] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 409.204257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 409.431755] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 409.531362] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 409.533995] LNet: Accept secure, port 988 [ 411.144138] Key type lgssc registered [ 411.551610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 415.345422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.549422] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.575427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.354788] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 421.857129] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.085561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.223290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.998789] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 427.083925] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 429.464412] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 432.255721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.393589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.448133] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 434.197734] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 437.956680] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 442.850186] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 442.850743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.851829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 442.861960] Lustre: Skipped 2 previous similar messages [ 444.394870] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 444.397976] Lustre: Skipped 2 previous similar messages [ 447.770100] Lustre: server umount lustre-MDT0000 complete [ 447.970471] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.979089] LustreError: Skipped 2 previous similar messages [ 449.582804] LustreError: 29106:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047065 with bad export cookie 5954223746329351892 [ 449.589673] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.089422] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.089475] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.098484] LustreError: Skipped 1 previous similar message [ 453.105304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 455.137512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 455.141090] Lustre: Skipped 2 previous similar messages [ 455.771905] Lustre: server umount lustre-MDT0001 complete [ 474.212172] Lustre: server umount lustre-OST0000 complete [ 482.859438] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 484.258536] Key type lgssc unregistered [ 484.392395] LNet: 33105:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.395987] LNet: Removed LNI 192.168.201.131@tcp [ 484.770950] Key type .llcrypt unregistered [ 484.772752] Key type ._llcrypt unregistered [ 495.330657] alg: No test for adler32 (adler32-zlib) [ 496.083414] Key type ._llcrypt registered [ 496.085148] Key type .llcrypt registered [ 496.133828] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 496.618248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 496.779901] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 496.866817] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 496.868816] LNet: Accept secure, port 988 [ 498.456142] Key type lgssc registered [ 498.938134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 502.582517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.746902] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.764848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.443993] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 508.900486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.239048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.392338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 511.213213] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 514.265642] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 516.703594] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 519.529619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.671368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.356437] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 523.753149] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 524.536767] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 528.866043] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.866175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.870232] LustreError: Skipped 1 previous similar message [ 528.878416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.882022] Lustre: Skipped 1 previous similar message [ 530.921313] Lustre: lustre-OST0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 533.985905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.993363] Lustre: Skipped 1 previous similar message [ 534.167018] Lustre: server umount lustre-OST0000 complete [ 549.345428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.348508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.350381] Lustre: Skipped 2 previous similar messages [ 554.464952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.468726] Lustre: Skipped 2 previous similar messages [ 562.144123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 562.190218] Lustre: server umount lustre-MDT0000 complete [ 563.751471] LustreError: 37214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047180 with bad export cookie 17322501754161384550 [ 563.753942] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.755838] LustreError: 37214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 578.016214] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 578.137066] Lustre: server umount lustre-MDT0001 complete [ 583.407931] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 584.603398] Key type lgssc unregistered [ 584.732387] LNet: 38395:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.735543] LNet: Removed LNI 192.168.201.131@tcp [ 585.066169] Key type .llcrypt unregistered [ 585.067862] Key type ._llcrypt unregistered [ 595.068536] alg: No test for adler32 (adler32-zlib) [ 595.820435] Key type ._llcrypt registered [ 595.821487] Key type .llcrypt registered [ 595.866727] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 596.208933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 596.478022] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 596.564945] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 596.567752] LNet: Accept secure, port 988 [ 598.164411] Key type lgssc registered [ 598.758444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 602.245504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.443866] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.462734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.175581] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 608.661746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.738988] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 608.817758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 610.404303] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 613.508707] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 615.976906] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 618.933225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.979518] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.085126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.895134] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 621.056775] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 623.857512] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 627.172622] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 628.840859] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 635.876635] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 635.879991] LNetError: 39156:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 635.883468] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 639.743214] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0919h20m55s [ 640.367856] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 644.577828] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.579075] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.580066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 644.583984] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 644.586769] Lustre: Skipped 2 previous similar messages [ 644.589595] LNetError: 39153:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 644.598287] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 645.608263] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 645.611223] Lustre: Skipped 2 previous similar messages [ 647.373369] Lustre: server umount lustre-MDT0000 complete [ 648.834408] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 649.697519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.701186] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 649.703241] LustreError: Skipped 2 previous similar messages [ 649.705752] LNetError: 39154:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 649.710701] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 651.500930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.552664] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.552753] LustreError: 11-0: MGC192.168.201.131@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 651.556625] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0x3c38f1303a7558d to 0x3c38f1303a759d1 [ 651.563033] Lustre: MGC192.168.201.131@tcp: Connection restored to 192.168.201.131@tcp (at 0@lo) [ 651.644903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.673125] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 653.265264] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 654.587836] Lustre: DEBUG MARKER: Wait for df (28821) ... [ 655.181189] Lustre: DEBUG MARKER: done [ 655.546668] LustreError: 39833:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.31@tcp arrived at 1781047271 with bad export cookie 271217714400811249 [ 655.552239] LustreError: 39833:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 656.865768] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 656.866547] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 656.866735] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 656.868384] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 656.869223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.871888] LNetError: 39155:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 656.875264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 656.877221] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 656.882419] Lustre: Skipped 1 previous similar message [ 661.985947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 661.988808] Lustre: Skipped 1 previous similar message [ 662.423080] Lustre: server umount lustre-OST0000 complete [ 667.105575] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 667.105663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.116568] Lustre: Skipped 2 previous similar messages [ 667.122435] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 667.128249] LNetError: 39156:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 667.131821] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 672.224900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 672.227664] Lustre: Skipped 3 previous similar messages [ 678.368287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 678.406462] Lustre: server umount lustre-MDT0000 complete [ 679.795136] LustreError: 39834:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047296 with bad export cookie 271217714400811473 [ 679.797156] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.799541] LustreError: 39834:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 684.914950] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 685.994570] Key type lgssc unregistered [ 686.117587] LNet: 44947:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.120866] LNet: Removed LNI 192.168.201.131@tcp [ 686.428813] Key type .llcrypt unregistered [ 686.430366] Key type ._llcrypt unregistered [ 699.294570] alg: No test for adler32 (adler32-zlib) [ 700.047316] Key type ._llcrypt registered [ 700.048622] Key type .llcrypt registered [ 700.093865] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 706.413073] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 706.849585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 707.007624] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 707.092577] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 707.094631] LNet: Accept secure, port 988 [ 708.680100] Key type lgssc registered [ 709.067103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 712.546968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 715.372190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 717.854474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 720.167086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 722.687535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 722.712988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.797552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 723.808686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 723.845556] Lustre: lustre-MDT0000: new disk, initializing [ 723.873543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.881809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 725.391431] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 730.134235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 730.158611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.194410] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 730.211790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 730.214093] Lustre: Skipped 1 previous similar message [ 730.251258] Lustre: lustre-MDT0001: new disk, initializing [ 730.294593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 730.309288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 730.313929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 732.095742] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 736.536431] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 738.854951] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 741.586814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 741.619210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.622911] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 741.720541] Lustre: lustre-OST0000: new disk, initializing [ 741.723100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 741.752152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.364278] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 747.002557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 747.007132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 747.767622] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 749.416902] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 749.501482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 751.440426] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 751.530624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 754.399824] Lustre: server umount lustre-OST0000 complete [ 756.195968] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.199096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.204313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.217150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.219736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.336148] Lustre: 52715:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047372/real 1781047372] req@000000005d82a7df x1867563477776064/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781047378 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 762.338280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.347166] Lustre: Skipped 2 previous similar messages [ 762.405964] Lustre: server umount lustre-MDT0000 complete [ 764.149515] LustreError: 49890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047380 with bad export cookie 17996145568391352265 [ 764.152505] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.155912] LustreError: 49890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.528176] Lustre: 52917:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047380/real 1781047380] req@000000009d450cd5 x1867563477777280/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781047386 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 770.656996] Lustre: server umount lustre-MDT0001 complete [ 774.351875] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 19:23:10 (1781047390) [ 777.155852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.333913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.360712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.029340] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 782.765967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.817051] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.941628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 784.622507] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 787.788594] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 790.250855] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 792.947817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 793.078134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.755202] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 797.781654] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 801.446381] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0919h23m37s [ 802.095201] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 804.939198] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 805.934196] Lustre: DEBUG MARKER: Creating a test file and stat it [ 805.977911] LustreError: 55534:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 807.910101] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 807.915270] LNetError: 47253:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 807.921836] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 826.353576] Lustre: lustre-MDT0000: Client 34affacc-2a8e-4072-aeff-5e71e012edc7 (at 192.168.201.31@tcp) reconnecting [ 846.024056] LustreError: 55534:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 846.881958] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 847.907564] Lustre: DEBUG MARKER: done [ 850.400964] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 850.404374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.413312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 854.498914] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 854.498966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 854.501792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.901694] Lustre: server umount lustre-OST0000 complete [ 859.618020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.619265] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 859.623513] Lustre: Skipped 1 previous similar message [ 859.629875] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 859.632849] LNetError: 47250:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 859.636078] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 863.753349] Lustre: server umount lustre-MDT0000 complete [ 864.739463] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.740775] LNetError: 120-3: Refusing connection from 192.168.201.131 for 127.0.0.2@tcp: No matching NI [ 864.745595] LustreError: Skipped 1 previous similar message [ 864.748674] LNetError: 47251:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 864.748718] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 865.190026] LustreError: 55537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047481 with bad export cookie 17996145568391353119 [ 865.192834] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.194020] LustreError: 55537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 870.539059] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 871.605144] Key type lgssc unregistered [ 871.722343] LNet: 57994:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.725825] LNet: Removed LNI 192.168.201.131@tcp [ 872.040310] Key type .llcrypt unregistered [ 872.041609] Key type ._llcrypt unregistered [ 885.360141] alg: No test for adler32 (adler32-zlib) [ 886.112401] Key type ._llcrypt registered [ 886.113789] Key type .llcrypt registered [ 886.164974] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 893.425455] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 893.756650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 894.009548] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 894.094209] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 894.095957] LNet: Accept secure, port 988 [ 895.688122] Key type lgssc registered [ 896.108809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 899.519346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 902.183968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 904.559820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 906.939452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 909.588044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 909.617311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.713587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 910.727482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 910.767300] Lustre: lustre-MDT0000: new disk, initializing [ 910.798164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.807385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 912.549756] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 917.459408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 917.481026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.521142] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 917.537912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 917.540352] Lustre: Skipped 1 previous similar message [ 917.581433] Lustre: lustre-MDT0001: new disk, initializing [ 917.620558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 917.636376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 917.639907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 919.215149] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 923.212925] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.255873] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 927.655185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 927.680446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 927.683660] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 927.772374] Lustre: lustre-OST0000: new disk, initializing [ 927.774971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 927.805305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.374505] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 933.534511] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 935.241825] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 936.441914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 936.445683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 937.366195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 938.931280] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 939.013610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 941.538233] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 941.538474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.541515] LustreError: Skipped 1 previous similar message [ 941.546783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 941.550726] Lustre: Skipped 1 previous similar message [ 945.812513] Lustre: server umount lustre-OST0000 complete [ 948.704974] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 948.708377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.713392] Lustre: Skipped 1 previous similar message [ 948.715774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.777094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.777511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.415349] Lustre: server umount lustre-MDT0000 complete [ 954.917438] LustreError: 62079:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047571 with bad export cookie 12874987206769672148 [ 954.919323] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.922603] LustreError: 62079:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 955.041702] Lustre: server umount lustre-MDT0001 complete [ 958.254529] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 19:26:14 (1781047574) [ 958.809453] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 959.402711] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 19:26:16 (1781047576) [ 963.494277] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 964.964635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.135385] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.155595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.520992] Lustre: Failing over lustre-MDT0000 [ 965.522927] LustreError: 67122:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 965.526457] LustreError: 67122:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 965.529103] LustreError: 67122:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 970.720133] Lustre: 67173:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047581/real 1781047581] req@00000000740612db x1867563673863040/t0(0) o251->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781047587 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 970.780514] Lustre: server umount lustre-MDT0000 complete [ 973.176261] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:26:29 (1781047589) [ 975.499046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.664881] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.679256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.247200] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 980.370595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.991388] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 984.753121] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 986.920109] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 989.289569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 989.392574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.396124] Lustre: Skipped 1 previous similar message [ 990.804162] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 993.458459] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1000.492730] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1006.050216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.051172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.056710] Lustre: Skipped 2 previous similar messages [ 1006.059322] Lustre: Skipped 2 previous similar messages [ 1006.062848] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1008.519193] Lustre: server umount lustre-MDT0000 complete [ 1009.896440] LustreError: 67719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047626 with bad export cookie 12874987206769673261 [ 1009.898928] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.900594] LustreError: 67719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1011.170131] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1012.764042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1015.622168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.225446] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0xb2ad24a66334842d to 0xb2ad24a663348bb2 [ 1016.230034] Lustre: MGC192.168.201.131@tcp: Connection restored to 192.168.201.131@tcp (at 0@lo) [ 1016.332071] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.358199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.917395] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1018.208089] Lustre: 60311:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047627/real 1781047627] req@00000000c7249c70 x1867563673877120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781047634 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1018.216277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.795439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.930151] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1021.924551] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1022.302438] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1022.944098] Lustre: 60309:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047632/real 1781047632] req@00000000e499f38f x1867563673877248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781047639 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1022.952663] Lustre: 60309:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1024.893722] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.917222] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1031.137320] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1031.140502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.144727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.146956] Lustre: Skipped 1 previous similar message [ 1034.831366] Lustre: server umount lustre-MDT0000 complete [ 1034.833398] Lustre: Skipped 1 previous similar message [ 1036.151955] LustreError: 70354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047652 with bad export cookie 12874987206769675186 [ 1036.155233] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1036.155858] LustreError: 70354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1038.661285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1041.956797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.337305] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0xb2ad24a663348bb2 to 0xb2ad24a6633491b6 [ 1042.342415] Lustre: MGC192.168.201.131@tcp: Connection restored to (at 0@lo) [ 1042.344375] Lustre: Skipped 1 previous similar message [ 1042.447502] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.452051] LustreError: Skipped 2 previous similar messages [ 1042.493534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1043.961365] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1044.513086] Lustre: 60308:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047653/real 1781047653] req@000000000b05598d x1867563673888320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781047660 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1047.023976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.183658] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1048.165521] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1048.699543] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1049.184063] Lustre: 60310:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047658/real 1781047658] req@000000006efcec6e x1867563673888448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781047665 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1049.191296] Lustre: 60310:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1051.333131] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1053.316231] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1057.761205] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1057.763981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.767980] Lustre: Skipped 3 previous similar messages [ 1057.769802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.772188] Lustre: Skipped 3 previous similar messages [ 1061.196815] Lustre: server umount lustre-MDT0000 complete [ 1061.198378] Lustre: Skipped 1 previous similar message [ 1062.488882] LustreError: 67719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047678 with bad export cookie 12874987206769676726 [ 1062.490411] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.492337] LustreError: 67719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1064.833365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1066.009309] Lustre: DEBUG MARKER: files compared the same [ 1073.120185] Lustre: 76857:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047683/real 1781047683] req@0000000056461180 x1867563673897920/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781047689 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1080.003808] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1080.915788] Key type lgssc unregistered [ 1081.024332] LNet: 77763:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.027766] LNet: Removed LNI 192.168.201.131@tcp [ 1081.316142] Key type .llcrypt unregistered [ 1081.317279] Key type ._llcrypt unregistered [ 1090.177327] alg: No test for adler32 (adler32-zlib) [ 1090.929454] Key type ._llcrypt registered [ 1090.931142] Key type .llcrypt registered [ 1090.972467] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1091.368863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1091.650738] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1091.728680] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1091.731730] LNet: Accept secure, port 988 [ 1093.312125] Key type lgssc registered [ 1093.659199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1096.540340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.706988] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.728902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.349555] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug +malloc all 8 [ 1102.707512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.821351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1104.482847] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug +malloc all 8 [ 1107.367613] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1109.569211] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1112.197865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.299076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.315765] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1113.898687] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug +malloc all 8 [ 1116.688544] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1118.436124] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1128.928926] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.929439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.932112] LustreError: Skipped 1 previous similar message [ 1128.937149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.940344] Lustre: Skipped 1 previous similar message [ 1130.452317] Lustre: server umount lustre-OST0000 complete [ 1133.536519] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1133.539200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.542893] Lustre: Skipped 1 previous similar message [ 1133.544754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.168451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.170922] Lustre: Skipped 3 previous similar messages [ 1144.288547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.293551] Lustre: Skipped 1 previous similar message [ 1146.336116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1146.374808] Lustre: server umount lustre-MDT0000 complete [ 1147.739614] LustreError: 79204:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047764 with bad export cookie 16079189306519782285 [ 1147.741027] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.744399] LustreError: 79204:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1147.885080] Lustre: server umount lustre-MDT0001 complete [ 1152.735527] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1153.771696] Key type lgssc unregistered [ 1153.890237] LNet: 82875:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.892433] LNet: Removed LNI 192.168.201.131@tcp [ 1154.204221] Key type .llcrypt unregistered [ 1154.205708] Key type ._llcrypt unregistered [ 1163.183219] alg: No test for adler32 (adler32-zlib) [ 1163.934444] Key type ._llcrypt registered [ 1163.935820] Key type .llcrypt registered [ 1163.974799] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1164.493421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1164.680393] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1164.771911] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1164.774524] LNet: Accept secure, port 988 [ 1166.360171] Key type lgssc registered [ 1166.740885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1169.484521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.976243] LustreError: 84307:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 1: rc = -5 [ 1217.696190] LustreError: 84307:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 4: rc = -110 [ 1248.416119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.862816] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1252.421401] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1255.215692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.256437] Lustre: *** cfs_fail_loc=706, val=0*** [ 1270.021172] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.046474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.066032] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1271.467064] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1274.416515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.540160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1274.574095] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1275.985368] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1278.576870] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1280.583021] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1285.089752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.094167] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.094327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.099228] Lustre: Skipped 1 previous similar message [ 1287.384825] Lustre: server umount lustre-OST0000 complete [ 1290.209988] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.210090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.210965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.219250] Lustre: Skipped 2 previous similar messages [ 1294.728852] Lustre: server umount lustre-MDT0000 complete [ 1295.328982] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.334862] LustreError: Skipped 1 previous similar message [ 1296.122049] LustreError: 87243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047912 with bad export cookie 16894014770190350584 [ 1296.126568] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.127710] LustreError: 87243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1296.254750] Lustre: server umount lustre-MDT0001 complete [ 1300.185368] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1301.201863] Key type lgssc unregistered [ 1301.317395] LNet: 88047:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.320488] LNet: Removed LNI 192.168.201.131@tcp [ 1301.601028] Key type .llcrypt unregistered [ 1301.602615] Key type ._llcrypt unregistered [ 1311.435787] alg: No test for adler32 (adler32-zlib) [ 1312.187383] Key type ._llcrypt registered [ 1312.188642] Key type .llcrypt registered [ 1312.227272] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1312.566140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.776102] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1312.858112] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1312.860344] LNet: Accept secure, port 988 [ 1314.440103] Key type lgssc registered [ 1314.819589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1317.574639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.100270] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1322.843654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.997960] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.037522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1324.492538] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1327.413460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.517310] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.521902] LustreError: Skipped 3 previous similar messages [ 1327.524942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.547279] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.553972] LustreError: Skipped 2 previous similar messages [ 1328.555204] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1328.992631] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1332.101086] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.105509] LustreError: Skipped 2 previous similar messages [ 1332.185968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.207445] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1333.620853] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1342.433840] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1342.436967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.441763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1342.447980] Lustre: Skipped 1 previous similar message [ 1347.554559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1347.556801] Lustre: Skipped 1 previous similar message [ 1348.118852] Lustre: server umount lustre-OST0000 complete [ 1352.674890] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1352.675274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.675978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.686212] Lustre: Skipped 2 previous similar messages [ 1355.648879] Lustre: server umount lustre-MDT0000 complete [ 1357.155836] Lustre: server umount lustre-MDT0001 complete [ 1363.936239] Lustre: 92908:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781047974/real 1781047974] req@000000009cfe69eb x1867564113210048/t0(0) o251->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781047980 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1363.950274] Lustre: server umount MGS complete [ 1368.061641] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:33:04 (1781047984) [ 1378.307547] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 1381.525747] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1384.980890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1387.289414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1389.308062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1391.408462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.700392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.720469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.177794] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1399.119631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1399.139630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.221108] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1399.236765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1399.274565] Lustre: lustre-MDT0001: new disk, initializing [ 1399.299753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.308364] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1399.311176] LustreError: 97601:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1400.717435] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1405.062055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1405.090180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1405.093500] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1405.188394] Lustre: lustre-OST0000: new disk, initializing [ 1405.191477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1405.191615] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1405.196627] Lustre: Skipped 1 previous similar message [ 1405.214895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.655262] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1410.591240] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1411.553578] Lustre: 98427:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000e3aa88b1 x1867564113214592/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781048039 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1412.886763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1412.900863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1412.939113] Lustre: lustre-MDT0000: new disk, initializing [ 1412.988128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1414.049661] Lustre: 99454:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000e560cafb x1867564113217728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781048042 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1414.419374] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1419.104464] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1420.643170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1420.647374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1423.138211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1423.140824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1443.809098] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.812117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.816731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.497037] Lustre: server umount lustre-OST0000 complete [ 1455.585339] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.585448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.588829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.593070] Lustre: Skipped 2 previous similar messages [ 1455.595106] Lustre: Skipped 2 previous similar messages [ 1465.312105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1465.347796] Lustre: server umount lustre-MDT0000 complete [ 1465.825142] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.829215] LustreError: Skipped 1 previous similar message [ 1480.160137] Lustre: 100652:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048090/real 1781048090] req@00000000d41464a0 x1867564113228352/t0(0) o251->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781048096 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1483.798770] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:35:00 (1781048100) [ 1490.898593] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1491.910134] Key type lgssc unregistered [ 1492.027358] LNet: 102056:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.030486] LNet: Removed LNI 192.168.201.131@tcp [ 1492.327310] Key type .llcrypt unregistered [ 1492.328748] Key type ._llcrypt unregistered [ 1499.149639] alg: No test for adler32 (adler32-zlib) [ 1499.901423] Key type ._llcrypt registered [ 1499.903092] Key type .llcrypt registered [ 1499.938540] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1500.328207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1500.493265] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1500.571449] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1500.574198] LNet: Accept secure, port 988 [ 1502.152101] Key type lgssc registered [ 1502.480656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1505.025500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.153098] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.166916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.645444] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1510.641759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.753589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.169221] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1514.798842] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.745298] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1519.231496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1519.345208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.741743] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1522.406144] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1523.351353] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1531.873874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.874368] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1531.878323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1531.882490] Lustre: Skipped 1 previous similar message [ 1536.858228] Lustre: server umount lustre-OST0000 complete [ 1541.600844] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1541.603715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.607817] Lustre: Skipped 1 previous similar message [ 1541.609624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.232760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.234573] Lustre: Skipped 3 previous similar messages [ 1552.352169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1552.392031] Lustre: server umount lustre-MDT0000 complete [ 1553.376842] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.380555] LustreError: Skipped 1 previous similar message [ 1553.543703] LustreError: 103301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048169 with bad export cookie 2599448268107524588 [ 1553.546411] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1553.548544] LustreError: 103301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1553.652475] Lustre: server umount lustre-MDT0001 complete [ 1558.175673] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1559.119869] Key type lgssc unregistered [ 1559.230320] LNet: 106976:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.233218] LNet: Removed LNI 192.168.201.131@tcp [ 1559.512796] Key type .llcrypt unregistered [ 1559.513967] Key type ._llcrypt unregistered [ 1565.725871] alg: No test for adler32 (adler32-zlib) [ 1566.477321] Key type ._llcrypt registered [ 1566.478318] Key type .llcrypt registered [ 1566.517270] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1566.864765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1567.051795] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1567.117832] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1567.119248] LNet: Accept secure, port 988 [ 1568.704103] Key type lgssc registered [ 1569.040420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.940530] LustreError: 108103:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1570.940660] LustreError: 108104:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1570.945055] LustreError: 108103:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4997 [ 1570.947328] LustreError: 108103:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1570.951165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.951675] LustreError: 108103:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1572.077897] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1572.090974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.557555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.657705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1576.039664] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1577.643258] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing lsmod [ 1578.542569] LustreError: 109410:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1578.542684] LustreError: 109411:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1578.548397] LustreError: 109410:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4996 [ 1578.550557] LustreError: 109410:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1578.553514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.555763] LustreError: 109410:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1578.662320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.665727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.666567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.669095] Lustre: Skipped 2 previous similar messages [ 1579.674093] Lustre: Skipped 2 previous similar messages [ 1585.121192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.123339] Lustre: Skipped 2 previous similar messages [ 1585.665968] Lustre: server umount lustre-MDT0000 complete [ 1586.877055] LustreError: 108117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048203 with bad export cookie 16775990682883446183 [ 1586.879756] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.880919] LustreError: 108117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1586.982860] Lustre: server umount lustre-MDT0001 complete [ 1594.336089] Lustre: 110007:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048204/real 1781048204] req@00000000d5fa0173 x1867564379545792/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781048210 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1594.400584] Lustre: server umount lustre-OST0000 complete [ 1597.068260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.197937] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.215653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.566603] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1601.267404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.367847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.660327] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1605.055485] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1606.962241] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1609.020125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.501072] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1612.197496] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1612.871135] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.474724] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1621.475564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.477825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.482107] Lustre: Skipped 1 previous similar message [ 1621.485192] Lustre: Skipped 1 previous similar message [ 1626.578061] Lustre: server umount lustre-OST0000 complete [ 1631.713287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.713708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1631.717196] Lustre: Skipped 1 previous similar message [ 1633.858068] Lustre: server umount lustre-MDT0000 complete [ 1635.104059] LustreError: 110462:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048251 with bad export cookie 16775990682883446869 [ 1635.106391] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.108935] LustreError: 110462:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1640.019589] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1640.991823] Key type lgssc unregistered [ 1641.099407] LNet: 114133:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.102579] LNet: Removed LNI 192.168.201.131@tcp [ 1641.399859] Key type .llcrypt unregistered [ 1641.401495] Key type ._llcrypt unregistered [ 1649.942297] alg: No test for adler32 (adler32-zlib) [ 1650.694311] Key type ._llcrypt registered [ 1650.695651] Key type .llcrypt registered [ 1650.730616] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1651.151739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1651.305461] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1651.378420] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1651.380473] LNet: Accept secure, port 988 [ 1652.960094] Key type lgssc registered [ 1653.278958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1655.745431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1656.866158] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.880472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.246009] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1661.006166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.121656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1662.554692] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1664.904953] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1666.725735] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1668.886983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1668.979946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.307345] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1672.038082] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1672.791150] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1679.702841] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1680.398082] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1681.377281] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.377534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.380424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.385496] Lustre: Skipped 1 previous similar message [ 1686.498554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1686.500950] Lustre: Skipped 2 previous similar messages [ 1687.374985] Lustre: server umount lustre-OST0000 complete [ 1691.617255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.617692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1691.620663] Lustre: Skipped 1 previous similar message [ 1694.661068] Lustre: server umount lustre-MDT0000 complete [ 1695.913973] LustreError: 118549:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048312 with bad export cookie 14598584614070569255 [ 1695.917142] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1695.917837] LustreError: 118549:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1696.038632] Lustre: server umount lustre-MDT0001 complete [ 1700.829068] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1701.863837] Key type lgssc unregistered [ 1701.988458] LNet: 119351:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.990942] LNet: Removed LNI 192.168.201.131@tcp [ 1702.301871] Key type .llcrypt unregistered [ 1702.303363] Key type ._llcrypt unregistered [ 1708.736380] alg: No test for adler32 (adler32-zlib) [ 1709.488335] Key type ._llcrypt registered [ 1709.489230] Key type .llcrypt registered [ 1709.522253] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1709.923051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.061768] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1710.137187] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1710.139228] LNet: Accept secure, port 988 [ 1711.720051] Key type lgssc registered [ 1712.019048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1714.409164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.530733] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.541818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.765268] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1719.483346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.614366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1720.964465] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1723.325822] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1725.129686] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1727.420168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1727.532221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.896132] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1731.435360] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1731.621993] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1740.769629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.770144] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1740.770483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.773957] Lustre: Skipped 1 previous similar message [ 1745.889990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.892438] Lustre: Skipped 2 previous similar messages [ 1746.128934] Lustre: server umount lustre-OST0000 complete [ 1750.496894] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1750.499805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1750.506751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.475607] Lustre: server umount lustre-MDT0000 complete [ 1754.629899] LustreError: 121403:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048370 with bad export cookie 4252795349207309854 [ 1754.632193] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.633306] LustreError: 121403:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.736744] Lustre: server umount lustre-MDT0001 complete [ 1759.091805] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1760.018667] Key type lgssc unregistered [ 1760.123909] LNet: 124267:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.127420] LNet: Removed LNI 192.168.201.131@tcp [ 1760.388385] Key type .llcrypt unregistered [ 1760.389642] Key type ._llcrypt unregistered [ 1768.705382] alg: No test for adler32 (adler32-zlib) [ 1769.457416] Key type ._llcrypt registered [ 1769.458557] Key type .llcrypt registered [ 1769.499307] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1769.989258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.187286] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1770.257357] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1770.258769] LNet: Accept secure, port 988 [ 1771.832059] Key type lgssc registered [ 1772.127273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1774.578780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.713509] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.729323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.132670] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1779.939712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.054212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1781.358314] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1783.713900] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1785.546450] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1787.756712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.888713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.307887] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1790.951324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1791.772848] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1798.969379] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1799.146636] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 1800.115942] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1800.118638] Lustre: Skipped 2 previous similar messages [ 1801.414717] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1801.416838] Lustre: Skipped 2 previous similar messages [ 1808.374383] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1808.383145] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1808.384850] Lustre: Skipped 1 previous similar message [ 1811.468227] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1811.470424] Lustre: Skipped 2 previous similar messages [ 1817.076220] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.31@tcp 192.168.0.[2,10]@tcp [ 1820.641945] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1820.642492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.643066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.649703] Lustre: Skipped 1 previous similar message [ 1825.175366] Lustre: server umount lustre-OST0000 complete [ 1830.881169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.881841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.884487] Lustre: Skipped 1 previous similar message [ 1830.886205] Lustre: Skipped 1 previous similar message [ 1832.450303] Lustre: server umount lustre-MDT0000 complete [ 1833.587958] LustreError: 128425:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048449 with bad export cookie 8839709446782138527 [ 1833.590055] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1833.591072] LustreError: 128425:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1833.700908] Lustre: server umount lustre-MDT0001 complete [ 1838.158962] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1839.137875] Key type lgssc unregistered [ 1839.242324] LNet: 130729:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.245029] LNet: Removed LNI 192.168.201.131@tcp [ 1839.498942] Key type .llcrypt unregistered [ 1839.499963] Key type ._llcrypt unregistered [ 1848.922734] alg: No test for adler32 (adler32-zlib) [ 1849.674319] Key type ._llcrypt registered [ 1849.675183] Key type .llcrypt registered [ 1849.708583] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1850.116631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.261590] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1850.337783] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1850.339506] LNet: Accept secure, port 988 [ 1851.920178] Key type lgssc registered [ 1852.224331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.635558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.766029] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.776987] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1855.779765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.784682] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.201.31@tcp 192.168.0.[2,10]@tcp [ 1857.078773] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1859.661897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.756445] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1859.764828] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.31@tcp 192.168.0.[2,10]@tcp [ 1859.776551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1861.090407] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1863.308265] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1864.986973] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1867.044128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1867.138658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.386882] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1870.594763] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1871.269299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1880.546361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.550784] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1880.551056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.556399] Lustre: Skipped 1 previous similar message [ 1885.665281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.667565] Lustre: Skipped 1 previous similar message [ 1885.910185] Lustre: server umount lustre-OST0000 complete [ 1890.785286] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1890.785459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.785922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.792938] Lustre: Skipped 2 previous similar messages [ 1893.248920] Lustre: server umount lustre-MDT0000 complete [ 1894.372920] LustreError: 132315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048510 with bad export cookie 1137401492290894554 [ 1894.375288] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1894.377221] LustreError: 132315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1894.487214] Lustre: server umount lustre-MDT0001 complete [ 1898.904639] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 1899.769700] Key type lgssc unregistered [ 1899.876298] LNet: 136037:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.878749] LNet: Removed LNI 192.168.201.131@tcp [ 1900.136683] Key type .llcrypt unregistered [ 1900.138440] Key type ._llcrypt unregistered [ 1909.001963] alg: No test for adler32 (adler32-zlib) [ 1909.753441] Key type ._llcrypt registered [ 1909.754440] Key type .llcrypt registered [ 1909.792875] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 1915.142341] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1915.610458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.760675] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1915.837431] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 1915.839246] LNet: Accept secure, port 988 [ 1917.416089] Key type lgssc registered [ 1917.748457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.629330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1922.768531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1924.746493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1926.692695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.832052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.852444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.929308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1929.939813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1929.972169] Lustre: lustre-MDT0000: new disk, initializing [ 1929.992789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.998349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1931.291421] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1935.228516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.253792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.286509] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1935.308528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1935.309557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1935.310565] Lustre: Skipped 1 previous similar message [ 1935.348699] Lustre: lustre-MDT0001: new disk, initializing [ 1935.376275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.387084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1935.390854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1936.716586] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1940.426914] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1942.387702] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1944.546292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1944.570264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1944.573030] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1944.665899] Lustre: lustre-OST0000: new disk, initializing [ 1944.667888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1944.696895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.079106] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1949.738718] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1950.196238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1950.199440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1951.078595] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1951.151658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1952.481801] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1952.554142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1955.297494] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1955.297598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.300309] LustreError: Skipped 1 previous similar message [ 1955.304426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1955.308012] Lustre: Skipped 1 previous similar message [ 1959.251377] Lustre: server umount lustre-OST0000 complete [ 1960.928819] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1960.931561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.935835] Lustre: Skipped 1 previous similar message [ 1960.937660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.536902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.537417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.594730] Lustre: server umount lustre-MDT0000 complete [ 1967.819026] LustreError: 139544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048584 with bad export cookie 5016334035699182881 [ 1967.821421] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.824275] LustreError: 139544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.932366] Lustre: server umount lustre-MDT0001 complete [ 1970.566510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.696591] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.715236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.060241] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1974.733585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.843031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.230837] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1978.710950] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1980.526174] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1982.796518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.893538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.236343] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1986.698897] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1988.113400] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1988.183567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1989.474625] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1990.562948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 1993.814293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1993.845613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.849171] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1993.897435] Lustre: lustre-OST0001: new disk, initializing [ 1993.899745] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1995.248954] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 1998.485101] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2001.401844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2001.404921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2001.616463] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2003.023124] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2003.093293] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2004.965575] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c7d1dfa000.ost_server_uuid 40 [ 2005.447162] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c7d1dfa000.ost_server_uuid in FULL state after 0 sec [ 2011.619107] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.623883] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.623932] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2011.628897] Lustre: Skipped 2 previous similar messages [ 2013.162105] Lustre: server umount lustre-OST0001 complete [ 2015.712983] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.714984] LustreError: Skipped 1 previous similar message [ 2015.716434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.719684] Lustre: Skipped 1 previous similar message [ 2016.740487] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.744266] LustreError: Skipped 1 previous similar message [ 2020.494166] Lustre: server umount lustre-OST0000 complete [ 2021.857940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.860065] Lustre: Skipped 4 previous similar messages [ 2029.003871] LustreError: 145856:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048645 with bad export cookie 5016334035699183735 [ 2029.006764] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.007315] LustreError: 145856:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.140575] Lustre: server umount lustre-MDT0001 complete [ 2029.142035] Lustre: Skipped 1 previous similar message [ 2036.283193] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 2037.174279] Key type lgssc unregistered [ 2037.273330] LNet: 150577:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.276410] LNet: Removed LNI 192.168.201.131@tcp [ 2037.550027] Key type .llcrypt unregistered [ 2037.551505] Key type ._llcrypt unregistered [ 2047.874939] alg: No test for adler32 (adler32-zlib) [ 2048.627466] Key type ._llcrypt registered [ 2048.628943] Key type .llcrypt registered [ 2048.660869] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2054.211207] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2054.621548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.748762] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2054.823185] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 2054.824717] LNet: Accept secure, port 988 [ 2056.400137] Key type lgssc registered [ 2056.700644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.436420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2061.470935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2063.285681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2065.191289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2067.262375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2067.278137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.348079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2068.357408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2068.390388] Lustre: lustre-MDT0000: new disk, initializing [ 2068.412152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.418653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2069.673424] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2073.694820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2073.713588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.740937] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2073.757525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2073.759761] Lustre: Skipped 1 previous similar message [ 2073.795907] Lustre: lustre-MDT0001: new disk, initializing [ 2073.825534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.836240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2073.839686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2075.121352] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2078.741175] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2080.542383] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2082.651088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2082.672130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.675504] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2082.753867] Lustre: lustre-OST0000: new disk, initializing [ 2082.755465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2082.776354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.048818] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2086.194052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2086.198211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2087.576689] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2088.925186] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2088.986318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2090.355891] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2090.437198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2091.489586] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.489770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.492958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.495707] Lustre: Skipped 1 previous similar message [ 2093.135456] Lustre: server umount lustre-OST0000 complete [ 2094.560748] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2094.563107] LustreError: Skipped 1 previous similar message [ 2094.564709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.570256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.572664] Lustre: Skipped 1 previous similar message [ 2096.608838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.609423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.422761] Lustre: server umount lustre-MDT0000 complete [ 2101.683070] LustreError: 154672:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048718 with bad export cookie 7640257976589698609 [ 2101.685094] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.686692] LustreError: 154672:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2101.791446] Lustre: server umount lustre-MDT0001 complete [ 2104.442616] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 19:45:21 (1781048721) [ 2109.398339] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2111.987479] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2115.044487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2117.125760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.949207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.889392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2122.988615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.008574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.085794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2123.096312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2123.136120] Lustre: lustre-MDT0000: new disk, initializing [ 2123.159242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.165409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2124.500733] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2128.393034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2128.412926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.443921] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2128.458584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2128.460343] Lustre: Skipped 1 previous similar message [ 2128.491263] Lustre: lustre-MDT0001: new disk, initializing [ 2128.516588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2128.525959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2128.528771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2129.730854] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2133.266675] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2134.925175] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2136.896073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2136.924762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.927262] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2137.000285] Lustre: lustre-OST0000: new disk, initializing [ 2137.001792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2137.023121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.294554] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2140.211205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2140.214327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2141.776264] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2143.664815] Lustre: Failing over lustre-OST0000 [ 2143.722771] Lustre: server umount lustre-OST0000 complete [ 2152.416131] Lustre: 152906:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048761/real 1781048761] req@000000001ee58af4 x1867564891267968/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781048768 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2152.416196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.422830] Lustre: 152906:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2152.429752] Lustre: Skipped 1 previous similar message [ 2155.488158] Lustre: 152905:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048764/real 1781048764] req@000000001db168cf x1867564891268480/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1781048771 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2156.781728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2156.895192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.901744] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2157.920202] Lustre: 152906:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048766/real 1781048766] req@00000000df24935e x1867564891268928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781048773 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2157.925244] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2158.340390] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2159.518354] Lustre: Failing over lustre-MDT0000 [ 2159.542954] LustreError: 152902:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2159.585583] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.587794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.592278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.594317] Lustre: Skipped 1 previous similar message [ 2159.685183] Lustre: server umount lustre-MDT0000 complete [ 2160.614230] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.631543] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.636209] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 2161.636417] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 2163.169054] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.172891] LustreError: Skipped 2 previous similar messages [ 2166.751468] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.208127] Lustre: 152906:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048779/real 1781048779] req@00000000f3e38367 x1867564891272832/t0(0) o400->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781048786 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2170.214778] Lustre: 152906:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2170.216715] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.512421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.895798] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.900027] LustreError: Skipped 7 previous similar messages [ 2176.481968] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0x6a07a6265ef94979 to 0x6a07a6265ef94e26 [ 2176.486492] Lustre: MGC192.168.201.131@tcp: Connection restored to (at 0@lo) [ 2176.534599] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 2176.570593] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2177.780852] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2178.016838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2181.603784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2181.610426] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2181.625558] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2186.721473] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.723265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.725184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2186.727803] Lustre: Skipped 3 previous similar messages [ 2186.730867] Lustre: Skipped 1 previous similar message [ 2188.629478] Lustre: server umount lustre-OST0000 complete [ 2191.842217] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2196.960948] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.965240] LustreError: Skipped 7 previous similar messages [ 2197.281923] LustreError: 162503:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048813 with bad export cookie 7640257976589700646 [ 2197.284562] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.286000] LustreError: 162503:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2201.757979] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 2202.624137] Key type lgssc unregistered [ 2202.720339] LNet: 167494:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.722824] LNet: Removed LNI 192.168.201.131@tcp [ 2202.981216] Key type .llcrypt unregistered [ 2202.982371] Key type ._llcrypt unregistered [ 2211.023022] alg: No test for adler32 (adler32-zlib) [ 2211.774377] Key type ._llcrypt registered [ 2211.775855] Key type .llcrypt registered [ 2211.805399] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2212.177170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.313213] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2212.381186] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 2212.383784] LNet: Accept secure, port 988 [ 2213.960120] Key type lgssc registered [ 2214.278560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.694296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.808759] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.821312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.008401] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2221.545774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.640817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.831613] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2225.053367] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2226.693457] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2228.632235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.720165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.029336] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2232.467377] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2232.805471] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2323.999180] Lustre: lustre-MDT0000: Client 909b53e0-a3a2-407c-9069-4d584579b9a5 (at 192.168.201.31@tcp) reconnecting [ 2332.680297] Lustre: lustre-MDT0000: Client 909b53e0-a3a2-407c-9069-4d584579b9a5 (at 192.168.201.31@tcp) reconnecting [ 2379.143385] Lustre: Failing over lustre-MDT0000 [ 2379.234148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.235328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.239918] Lustre: Skipped 2 previous similar messages [ 2379.248257] Lustre: Skipped 2 previous similar messages [ 2379.276416] Lustre: server umount lustre-MDT0000 complete [ 2383.856523] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.961572] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.970268] LustreError: Skipped 3 previous similar messages [ 2391.520597] Lustre: 168267:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781049000/real 1781049000] req@000000004e6d7639 x1867565056188544/t0(0) o400->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781049007 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2391.538059] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.547600] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.555471] LustreError: Skipped 6 previous similar messages [ 2393.711197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.130427] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.140416] LustreError: Skipped 1 previous similar message [ 2397.666285] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0xbb267567eb0789c8 to 0xbb267567eb09f5e2 [ 2397.672421] Lustre: MGC192.168.201.131@tcp: Connection restored to (at 0@lo) [ 2397.782100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.819324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2399.667878] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2401.249808] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2402.791146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2402.799615] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2402.822313] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 2405.805738] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.627490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2423.266246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.266780] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2423.267069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.273038] Lustre: Skipped 2 previous similar messages [ 2426.255929] Lustre: server umount lustre-MDT0000 complete [ 2427.885049] LustreError: 168933:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049044 with bad export cookie 13485595223392908770 [ 2427.887832] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.890652] LustreError: 168933:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2428.384813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.385279] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.388980] LustreError: Skipped 6 previous similar messages [ 2428.395730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2428.400954] Lustre: Skipped 2 previous similar messages [ 2432.417341] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2439.649064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2439.651351] Lustre: Skipped 1 previous similar message [ 2442.208130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2442.306891] Lustre: server umount lustre-MDT0001 complete [ 2450.144893] Lustre: server umount lustre-OST0000 complete [ 2454.985472] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2458.903698] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2462.952108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2465.637898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2467.736750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2469.860944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2472.315423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2472.341936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.435938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2472.449047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2472.490223] Lustre: lustre-MDT0000: new disk, initializing [ 2472.523224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.531989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2474.134303] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2478.859541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2478.887862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.926532] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2478.944623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2478.946936] Lustre: Skipped 1 previous similar message [ 2478.983873] Lustre: lustre-MDT0001: new disk, initializing [ 2479.034525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2479.040377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2480.805345] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2485.180093] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2487.279198] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2489.745322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2489.773312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.776283] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2489.874784] Lustre: lustre-OST0000: new disk, initializing [ 2489.876971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2489.903079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.905434] Lustre: Skipped 1 previous similar message [ 2491.545763] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2495.712836] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2497.613705] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2499.594204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2499.598970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2499.757613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2501.647305] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2501.750936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2504.675835] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.677887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.678742] LustreError: Skipped 1 previous similar message [ 2504.684227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.689204] Lustre: Skipped 1 previous similar message [ 2504.696068] Lustre: Skipped 1 previous similar message [ 2508.769656] Lustre: server umount lustre-OST0000 complete [ 2514.915154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.920904] Lustre: Skipped 1 previous similar message [ 2521.057860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2521.063314] Lustre: Skipped 5 previous similar messages [ 2525.154815] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2525.210916] Lustre: server umount lustre-MDT0000 complete [ 2526.177196] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.184730] LustreError: Skipped 4 previous similar messages [ 2527.178560] LustreError: 175811:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049143 with bad export cookie 13485595223392909491 [ 2527.181599] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.187706] LustreError: 175811:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2531.616994] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:52:28 (1781049148) [ 2539.362104] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2543.425376] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2548.088868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.000759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2553.737120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2556.289727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.129759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.164280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.264684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2559.281321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2559.323357] Lustre: lustre-MDT0000: new disk, initializing [ 2559.356718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.366584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2561.423272] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2566.805430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.844252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.907097] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2566.936455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2566.939502] Lustre: Skipped 1 previous similar message [ 2566.984238] Lustre: lustre-MDT0001: new disk, initializing [ 2567.039713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2567.044172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2569.083379] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2573.835737] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2576.365618] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2579.001397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2579.027845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.030793] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2579.124813] Lustre: lustre-OST0000: new disk, initializing [ 2579.127461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2580.866779] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2584.574334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2584.578162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2584.990906] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2589.666304] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2589.667472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.668819] LustreError: Skipped 1 previous similar message [ 2589.674280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2589.677682] Lustre: Skipped 1 previous similar message [ 2593.236464] Lustre: server umount lustre-OST0000 complete [ 2593.238243] Lustre: Skipped 1 previous similar message [ 2601.908852] LustreError: 184898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049218 with bad export cookie 13485595223392910331 [ 2601.910966] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.912389] LustreError: 184898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2604.765379] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:53:41 (1781049221) [ 2610.011862] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2613.079342] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2616.346729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.614710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2620.763195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2622.991820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2625.225733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2625.249539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.329594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2625.339909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2625.373026] Lustre: lustre-MDT0000: new disk, initializing [ 2625.394271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.396572] Lustre: Skipped 2 previous similar messages [ 2625.402488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2626.797185] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2630.863178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.888280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.011663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2632.433094] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2636.341191] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2638.448187] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2640.730521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.751401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.754688] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2642.209992] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2645.879593] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2650.104075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2655.202461] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2655.202887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.204958] LustreError: Skipped 1 previous similar message [ 2655.208300] Lustre: Skipped 3 previous similar messages [ 2655.208991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.214767] Lustre: Skipped 5 previous similar messages [ 2659.044367] Lustre: server umount lustre-OST0000 complete [ 2659.045536] Lustre: Skipped 2 previous similar messages [ 2667.616916] LustreError: 190119:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049283 with bad export cookie 13485595223392911395 [ 2667.619073] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.620808] LustreError: 190119:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2672.121640] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 2672.966032] Key type lgssc unregistered [ 2673.068298] LNet: 193222:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.070578] LNet: Removed LNI 192.168.201.131@tcp [ 2673.321924] Key type .llcrypt unregistered [ 2673.323059] Key type ._llcrypt unregistered [ 2681.561670] alg: No test for adler32 (adler32-zlib) [ 2682.313299] Key type ._llcrypt registered [ 2682.314177] Key type .llcrypt registered [ 2682.344848] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2682.698917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2682.818092] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2682.880884] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 2682.882248] LNet: Accept secure, port 988 [ 2684.448100] Key type lgssc registered [ 2684.744431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.038857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.156092] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.167655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.418794] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2692.141059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.241929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.554102] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2695.878236] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.625376] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2699.715971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.819613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.187175] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2702.893147] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2703.620875] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2722.274206] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2722.274663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.280667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2722.282328] Lustre: Skipped 1 previous similar message [ 2726.227579] Lustre: server umount lustre-OST0000 complete [ 2728.416651] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.418819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.422173] Lustre: Skipped 1 previous similar message [ 2728.424237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.512748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.513222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.570880] Lustre: server umount lustre-MDT0000 complete [ 2734.727051] LustreError: 194664:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049351 with bad export cookie 16265728134403479911 [ 2734.729750] LustreError: 194664:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2734.729953] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.837911] Lustre: server umount lustre-MDT0001 complete [ 2739.207307] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 2740.045556] Key type lgssc unregistered [ 2740.154395] LNet: 198354:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2740.157689] LNet: Removed LNI 192.168.201.131@tcp [ 2740.412705] Key type .llcrypt unregistered [ 2740.414030] Key type ._llcrypt unregistered [ 2748.825391] alg: No test for adler32 (adler32-zlib) [ 2749.576514] Key type ._llcrypt registered [ 2749.578383] Key type .llcrypt registered [ 2749.619395] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2750.066630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2750.212963] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2750.293583] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 2750.295189] LNet: Accept secure, port 988 [ 2751.872126] Key type lgssc registered [ 2752.204499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2754.752972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.912368] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.929673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.349669] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2760.182457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.301774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.782993] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2764.415371] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2766.396179] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2768.565377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2768.667525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.022107] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2770.725061] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2772.503562] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2780.129903] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2780.130383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.137276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2784.229328] Lustre: lustre-OST0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 2784.233087] Lustre: Skipped 1 previous similar message [ 2785.249427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.877116] Lustre: server umount lustre-OST0000 complete [ 2788.365379] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92c7c3fd4000.ost_server_uuid 40 [ 2788.817103] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92c7c3fd4000.ost_server_uuid in DISCONN state after 0 sec [ 2789.301634] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2800.609177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.609528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.612434] Lustre: Skipped 2 previous similar messages [ 2800.615149] Lustre: Skipped 2 previous similar messages [ 2805.312400] Lustre: server umount lustre-MDT0000 complete [ 2805.729082] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.734192] LustreError: Skipped 1 previous similar message [ 2806.464838] LustreError: 199797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049422 with bad export cookie 4688836280026667335 [ 2806.467054] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.469409] LustreError: 199797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2806.575650] Lustre: server umount lustre-MDT0001 complete [ 2809.173254] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:57:05 (1781049425) [ 2811.187890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.333285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.352085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.592063] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2815.195436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.522452] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2818.763818] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2820.482451] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2822.485781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.599242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.601787] Lustre: Skipped 1 previous similar message [ 2823.870587] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2825.702269] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2826.192713] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.393722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2828.417388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.421323] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2828.474852] Lustre: lustre-OST0001: new disk, initializing [ 2828.476521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2829.753128] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2831.423296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2831.427510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2833.389218] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2835.937072] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2835.937304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.939905] LustreError: Skipped 1 previous similar message [ 2835.940395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2835.943709] Lustre: Skipped 1 previous similar message [ 2840.298063] Lustre: server umount lustre-OST0000 complete [ 2841.572556] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.578885] LustreError: Skipped 1 previous similar message [ 2842.189024] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2842.256081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2843.532542] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2843.600513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2846.689567] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.693042] LustreError: Skipped 2 previous similar messages [ 2856.929771] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.929973] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2856.930148] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.931886] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2856.931891] Lustre: Skipped 4 previous similar messages [ 2856.935547] LustreError: Skipped 4 previous similar messages [ 2856.938651] LustreError: Skipped 1 previous similar message [ 2859.608221] Lustre: server umount lustre-OST0001 complete [ 2861.536967] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2861.540893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.547115] Lustre: Skipped 1 previous similar message [ 2873.313118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.315120] Lustre: Skipped 9 previous similar messages [ 2875.360110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2875.401861] Lustre: server umount lustre-MDT0000 complete [ 2876.606907] LustreError: 208248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049492 with bad export cookie 4688836280026668385 [ 2876.609558] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2876.610676] LustreError: 208248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2878.527105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2879.777315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2881.031147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2882.455836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2884.624390] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:58:21 (1781049501) [ 2886.768517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.858850] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2886.873048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2886.941024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.943724] Lustre: Skipped 1 previous similar message [ 2888.297202] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2891.161042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.194312] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2891.208675] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2892.770974] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2895.320951] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2897.280585] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2899.657178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.724334] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2901.084755] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2903.386877] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2905.498379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2905.517909] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2905.577911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2905.579869] Lustre: Skipped 2 previous similar messages [ 2906.897274] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2909.213503] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2909.296152] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 2909.306491] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 2914.273308] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2914.276539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.280841] Lustre: Skipped 1 previous similar message [ 2914.282937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.140901] Lustre: server umount lustre-OST0000 complete [ 2916.142309] Lustre: Skipped 1 previous similar message [ 2919.904738] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.912423] LustreError: Skipped 2 previous similar messages [ 2929.633092] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2929.636945] LustreError: Skipped 1 previous similar message [ 2935.265530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.269576] Lustre: Skipped 4 previous similar messages [ 2941.419652] LustreError: 210217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049557 with bad export cookie 4688836280026669750 [ 2941.423828] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.424068] LustreError: 210217:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2943.501973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.756982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2946.030516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.380869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2949.639869] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:59:26 (1781049566) [ 2954.941219] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2957.870708] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2961.296507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.467935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2965.428499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2967.448365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2969.763179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2969.787171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.876181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2969.885996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2969.923833] Lustre: lustre-MDT0000: new disk, initializing [ 2969.945516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.952155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2971.321069] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2975.485925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.512032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.540231] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2975.555258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2975.557246] Lustre: Skipped 1 previous similar message [ 2975.591396] Lustre: lustre-MDT0001: new disk, initializing [ 2975.631627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2975.634326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2976.976646] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2980.807183] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2982.730851] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2985.004040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2985.026159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.030947] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2985.116408] Lustre: lustre-OST0000: new disk, initializing [ 2985.118199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2986.473971] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 2986.805376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2986.809285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2990.220638] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2991.659695] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2991.731229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2993.145752] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2993.234937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2997.220474] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2997.220870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.224247] LustreError: Skipped 2 previous similar messages [ 2997.231007] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.237478] Lustre: Skipped 8 previous similar messages [ 3000.040497] Lustre: server umount lustre-OST0000 complete [ 3000.042798] Lustre: Skipped 3 previous similar messages [ 3007.456985] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.462342] LustreError: Skipped 8 previous similar messages [ 3008.647522] LustreError: 219269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049624 with bad export cookie 4688836280026671374 [ 3008.650952] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.650957] LustreError: 219269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3011.603614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.171908] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3015.987751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.601674] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3020.095734] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3021.951485] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3024.100941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.546057] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3027.944628] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3029.351099] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3029.428956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3030.815466] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3031.921420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3036.646560] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3040.744283] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3040.823247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3042.196750] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3065.761064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.866205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.869492] Lustre: Skipped 5 previous similar messages [ 3067.218317] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3068.963307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3068.968740] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 3068.968748] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3068.970827] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 3068.981932] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3069.712513] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3073.368196] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3079.138754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.144722] Lustre: Skipped 6 previous similar messages [ 3080.533259] Lustre: server umount lustre-OST0000 complete [ 3080.535385] Lustre: Skipped 3 previous similar messages [ 3089.224094] LustreError: 222754:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049705 with bad export cookie 4688836280026672214 [ 3089.227448] LustreError: 222754:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3089.227672] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.227141] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:01:48 (1781049708) [ 3094.433910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.586135] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.592875] LustreError: Skipped 1 previous similar message [ 3096.116800] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3098.919749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.361793] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3102.740500] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3104.425765] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3106.414521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.780346] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3110.073420] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3111.405571] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3111.473461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3112.757814] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3113.854722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3115.905413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3115.927609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.931718] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3115.984730] Lustre: lustre-OST0001: new disk, initializing [ 3115.986964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3117.265806] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3120.882540] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3121.130163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3121.134760] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3122.358236] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3122.444137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3123.896286] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3123.975939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3126.242095] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3126.245892] LustreError: Skipped 3 previous similar messages [ 3126.248906] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3126.250777] Lustre: Skipped 10 previous similar messages [ 3132.692047] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3132.762742] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3134.159693] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3134.229751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3157.768312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.226221] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3161.717222] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3162.237542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3195.360075] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3204.189925] LustreError: 228976:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049820 with bad export cookie 4688836280026673600 [ 3204.192408] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3204.194876] LustreError: 228976:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3210.378666] Lustre: server umount lustre-MDT0001 complete [ 3210.379991] Lustre: Skipped 6 previous similar messages [ 3212.261038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.624110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3214.889606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.152171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.067743] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:03:54 (1781049834) [ 3220.157640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.244647] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3220.255184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3220.319705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.322141] Lustre: Skipped 5 previous similar messages [ 3221.622627] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3224.298033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.325241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3225.780392] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3228.233756] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3230.369250] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3232.648951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.716060] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3234.150970] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3236.649621] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3240.319931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3241.710555] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3244.138440] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3244.699078] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3246.154071] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3246.229683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3247.681861] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3247.761541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3249.633340] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c7c4836000.ost_server_uuid 40 [ 3250.182035] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c7c4836000.ost_server_uuid in FULL state after 0 sec [ 3250.522350] Lustre: Permanently deactivating lustre-OST0001 [ 3250.527554] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3250.529840] Lustre: Skipped 1 previous similar message [ 3251.331935] Lustre: Permanently reactivating lustre-OST0001 [ 3253.220122] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.225834] Lustre: Skipped 9 previous similar messages [ 3260.897064] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.900418] LustreError: Skipped 4 previous similar messages [ 3263.459790] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.464300] LustreError: Skipped 31 previous similar messages [ 3281.376105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3282.742275] LustreError: 240847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049899 with bad export cookie 4688836280026674853 [ 3282.746076] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.747454] LustreError: 240847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3284.889683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3286.607085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3287.854896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3289.156764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3291.550383] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:05:08 (1781049908) [ 3291.966984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3294.370467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.470704] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3294.480153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3294.481751] Lustre: Skipped 5 previous similar messages [ 3295.910565] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3298.896060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.954447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3298.957714] Lustre: Skipped 1 previous similar message [ 3300.550612] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3302.982558] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3304.972661] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3307.668619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3307.691117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3307.763197] Lustre: Permanently deactivating lustre-OST0000 [ 3309.153437] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3310.769326] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3310.772110] LustreError: 247244:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 3311.638445] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3313.960893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.494939] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3318.275858] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3320.365678] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3324.279820] Lustre: Permanently reactivating lustre-OST0000 [ 3328.251738] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3328.254937] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3328.259917] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3328.262431] Lustre: Skipped 1 previous similar message [ 3328.275530] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3328.283534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3356.191851] LustreError: 246725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049972 with bad export cookie 4688836280026676435 [ 3356.194413] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.196228] LustreError: 246725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3361.001670] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 3361.950617] Key type lgssc unregistered [ 3362.056315] LNet: 249681:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3362.058834] LNet: Removed LNI 192.168.201.131@tcp [ 3362.348377] Key type .llcrypt unregistered [ 3362.350250] Key type ._llcrypt unregistered [ 3370.580769] alg: No test for adler32 (adler32-zlib) [ 3371.332286] Key type ._llcrypt registered [ 3371.333290] Key type .llcrypt registered [ 3371.374519] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3371.935570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3372.100427] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3372.187904] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 3372.190352] LNet: Accept secure, port 988 [ 3373.768088] Key type lgssc registered [ 3374.145554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3375.310324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3377.423163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.568843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.587722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.946580] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3382.836716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3382.866031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.898074] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3382.909078] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3382.911612] Lustre: Permanently deactivating lustre-MDT0001 [ 3383.010737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3384.301649] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3384.311848] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3386.895650] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3388.735177] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3390.844774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.946098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.249609] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3394.019340] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.023538] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 3394.706641] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.974322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3397.035339] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3398.056802] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 3398.057412] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 [ 3398.090785] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 3398.372349] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3400.730090] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3402.296589] Lustre: Permanently reactivating lustre-MDT0001 [ 3402.301907] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3402.304218] Lustre: Skipped 3 previous similar messages [ 3409.915761] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.921325] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3409.924849] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3409.929349] LustreError: 254786:0:(llog_osd.c:2133:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3409.929905] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3409.933492] LustreError: 254786:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3409.939101] LustreError: 254786:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.131@tcp: cfg command failed: rc = -108 [ 3409.943158] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3409.943158] [ 3409.947899] LustreError: 251184:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3410.405919] Lustre: lustre-MDT0001: Client bf4c9b8c-9f0f-4628-9b6f-46e171b09239 (at 192.168.201.31@tcp) reconnecting [ 3412.442069] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3413.122789] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3415.337172] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 3415.490466] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3416.905702] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 3416.995251] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3417.830980] Lustre: Permanently deactivating lustre-MDT0001 [ 3417.837711] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3417.839609] Lustre: Skipped 2 previous similar messages [ 3418.070986] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3427.812198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.813596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.815564] Lustre: Skipped 2 previous similar messages [ 3427.818227] Lustre: Skipped 3 previous similar messages [ 3432.522465] Lustre: server umount lustre-MDT0000 complete [ 3432.930348] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.934747] LustreError: Skipped 3 previous similar messages [ 3433.787602] LustreError: 253161:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050050 with bad export cookie 5662850657772811134 [ 3433.790872] LustreError: 253161:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3433.791489] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.910054] Lustre: server umount lustre-MDT0001 complete [ 3441.194559] Lustre: server umount lustre-OST0000 complete [ 3444.755791] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:07:41 (1781050061) [ 3448.336435] Lustre: server umount lustre-OST0001 complete [ 3450.560115] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3453.509616] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3456.546434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3458.591450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.527278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3462.558622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3464.735939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3464.759694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.844740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3464.846896] Lustre: Skipped 2 previous similar messages [ 3464.856093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3464.892637] Lustre: lustre-MDT0000: new disk, initializing [ 3464.916129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.922420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3466.297528] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3470.425258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3470.445839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.476390] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3470.494025] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3470.495815] Lustre: Skipped 1 previous similar message [ 3470.534239] Lustre: lustre-MDT0001: new disk, initializing [ 3470.576781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3470.580822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3472.024897] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3475.861423] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3477.725397] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3479.900711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.926557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3479.931618] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3480.021276] Lustre: lustre-OST0000: new disk, initializing [ 3480.022967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3480.046413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.049534] Lustre: Skipped 1 previous similar message [ 3480.112709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3480.115419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3481.406519] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3485.245159] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3485.907378] LustreError: 259665:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3490.008849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3490.032780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.035218] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3490.082798] Lustre: lustre-OST0001: new disk, initializing [ 3490.084585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3491.493467] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3495.224065] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3496.000112] LustreError: 259665:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3496.007176] LustreError: 259665:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3496.756053] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3497.466268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3497.470847] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3506.096129] LustreError: 259665:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3507.684344] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3507.689989] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.702671] Lustre: Skipped 1 previous similar message [ 3507.705426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3507.709474] Lustre: Skipped 1 previous similar message [ 3510.771795] Lustre: lustre-OST0001: Not available for connect from 192.168.201.31@tcp (stopping) [ 3512.826420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3513.403942] Lustre: server umount lustre-OST0001 complete [ 3515.984626] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.060118] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.069137] LustreError: Skipped 3 previous similar messages [ 3531.233883] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3531.235045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.244951] LustreError: Skipped 2 previous similar messages [ 3531.255281] Lustre: Skipped 1 previous similar message [ 3531.258133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3531.267138] Lustre: Skipped 2 previous similar messages [ 3533.289354] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.303765] LustreError: Skipped 3 previous similar messages [ 3543.530778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.535127] Lustre: Skipped 5 previous similar messages [ 3545.056772] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3545.185475] Lustre: server umount lustre-OST0000 complete [ 3548.642497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.652775] Lustre: Skipped 2 previous similar messages [ 3557.596492] LustreError: 260508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050173 with bad export cookie 5662850657772813871 [ 3557.600067] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.610601] LustreError: 260508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3563.960387] Lustre: server umount lustre-MDT0001 complete [ 3563.962085] Lustre: Skipped 1 previous similar message [ 3582.412899] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 3590.062879] Key type lgssc unregistered [ 3590.825101] LNet: 264872:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3590.851242] LNet: Removed LNI 192.168.201.131@tcp [ 3593.016246] Key type .llcrypt unregistered [ 3593.032276] Key type ._llcrypt unregistered [ 3642.359791] alg: No test for adler32 (adler32-zlib) [ 3643.115720] Key type ._llcrypt registered [ 3643.118842] Key type .llcrypt registered [ 3643.233307] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 3655.432503] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3656.745337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3657.119512] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3657.359543] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 3657.364255] LNet: Accept secure, port 988 [ 3659.024118] Key type lgssc registered [ 3659.771330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3666.863692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3672.651595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3678.166415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3683.747038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3690.289704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3690.340653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.561293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3691.587773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3691.649456] Lustre: lustre-MDT0000: new disk, initializing [ 3691.720618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.738782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3695.686642] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3706.050786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3706.126295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.188903] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3706.212047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3706.214868] Lustre: Skipped 1 previous similar message [ 3706.317150] Lustre: lustre-MDT0001: new disk, initializing [ 3706.414435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.457079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3706.470205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3710.647443] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3719.846638] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3724.888228] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3731.423544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3731.481155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.489679] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3731.705511] Lustre: lustre-OST0000: new disk, initializing [ 3731.709821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3731.786811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.011896] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3740.211811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3740.227012] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3745.642200] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3750.008968] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3750.226441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3754.637618] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3754.836173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3761.125594] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.127796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.139779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.148697] Lustre: Skipped 1 previous similar message [ 3761.164916] Lustre: Skipped 1 previous similar message [ 3762.737400] Lustre: server umount lustre-OST0000 complete [ 3766.243222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.246102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3766.256507] Lustre: Skipped 1 previous similar message [ 3771.361657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.367540] Lustre: Skipped 2 previous similar messages [ 3771.910455] Lustre: server umount lustre-MDT0000 complete [ 3775.195327] LustreError: 268967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050391 with bad export cookie 14520819537926116324 [ 3775.200359] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3775.205709] LustreError: 268967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3775.664526] Lustre: server umount lustre-MDT0001 complete [ 3783.405045] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 20:13:19 (1781050399) [ 3789.927760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.540416] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3790.608388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.121716] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3795.941751] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.065660] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.479829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.794202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3808.710331] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3816.281346] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3821.609899] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3827.672318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.868608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.742416] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3837.902960] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3854.818838] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3854.849239] LustreError: Skipped 1 previous similar message [ 3854.857922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.873363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.869183] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3856.875230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.791348] Lustre: server umount lustre-OST0000 complete [ 3862.608453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3870.833600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3871.103984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.140140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3873.154899] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 3873.158611] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3874.258909] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3878.321821] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3878.330058] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3878.337946] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3879.522862] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3888.618375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.619048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.622423] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.635530] Lustre: Skipped 1 previous similar message [ 3888.638139] Lustre: Skipped 2 previous similar messages [ 3891.314885] Lustre: server umount lustre-OST0000 complete [ 3898.849126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.860881] Lustre: Skipped 4 previous similar messages [ 3899.843569] Lustre: server umount lustre-MDT0000 complete [ 3902.447769] LustreError: 273507:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050518 with bad export cookie 14520819537926117178 [ 3902.451568] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.456161] LustreError: 273507:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3910.409185] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 3912.394176] Key type lgssc unregistered [ 3912.553909] LNet: 278940:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.559632] LNet: Removed LNI 192.168.201.131@tcp [ 3913.032809] Key type .llcrypt unregistered [ 3913.035211] Key type ._llcrypt unregistered [ 3926.665027] hrtimer: interrupt took 7017330 ns [ 3927.777108] alg: No test for adler32 (adler32-zlib) [ 3928.528448] Key type ._llcrypt registered [ 3928.530573] Key type .llcrypt registered [ 3928.611769] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3929.341925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3929.541677] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3929.670533] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 3929.675525] LNet: Accept secure, port 988 [ 3931.272105] Key type lgssc registered [ 3931.841531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3935.976954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.197542] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.222070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.381635] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3942.370526] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.341802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.579764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3946.588682] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3950.674893] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3953.677266] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3957.211046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3957.384564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.813650] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 3964.020500] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3967.676722] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 3980.260180] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.270231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.278514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3981.910140] Lustre: server umount lustre-OST0000 complete [ 3988.449338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.449972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.454813] Lustre: Skipped 1 previous similar message [ 3990.496267] Lustre: 284022:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781050600/real 1781050600] req@00000000a2d2aad5 x1867566856286784/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781050606 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3990.646430] Lustre: server umount lustre-MDT0000 complete [ 3992.789091] LustreError: 280382:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050609 with bad export cookie 1631078211144856898 [ 3992.793662] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.800792] LustreError: 280382:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3992.967437] Lustre: server umount lustre-MDT0001 complete [ 4000.135455] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4001.923363] Key type lgssc unregistered [ 4002.124058] LNet: 284826:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4002.144567] LNet: Removed LNI 192.168.201.131@tcp [ 4002.673816] Key type .llcrypt unregistered [ 4002.677606] Key type ._llcrypt unregistered [ 4012.700309] alg: No test for adler32 (adler32-zlib) [ 4013.452446] Key type ._llcrypt registered [ 4013.454429] Key type .llcrypt registered [ 4013.506560] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4014.173816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4014.339398] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4014.445704] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4014.451299] LNet: Accept secure, port 988 [ 4016.056089] Key type lgssc registered [ 4016.573700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4020.281049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.473350] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.503304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.612062] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4026.851789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.204109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.388944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4030.758909] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4035.359695] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4038.489623] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.514937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.726384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.989328] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4048.825571] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4051.943703] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 4060.134314] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4060.142158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.150063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4062.177372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4062.180875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.893154] Lustre: server umount lustre-OST0000 complete [ 4067.298290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.300517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.303772] Lustre: Skipped 1 previous similar message [ 4067.307685] Lustre: Skipped 1 previous similar message [ 4072.417650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.419595] Lustre: Skipped 2 previous similar messages [ 4073.171215] Lustre: server umount lustre-MDT0000 complete [ 4075.554550] LustreError: 289111:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050691 with bad export cookie 8036945180523670647 [ 4075.559868] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4075.560972] LustreError: 289111:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4075.794548] Lustre: server umount lustre-MDT0001 complete [ 4082.423699] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4083.725478] Key type lgssc unregistered [ 4083.866675] LNet: 289914:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4083.871321] LNet: Removed LNI 192.168.201.131@tcp [ 4084.356922] Key type .llcrypt unregistered [ 4084.362372] Key type ._llcrypt unregistered [ 4094.453661] alg: No test for adler32 (adler32-zlib) [ 4095.206487] Key type ._llcrypt registered [ 4095.208427] Key type .llcrypt registered [ 4095.279454] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4095.870775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4096.090991] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4096.236771] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4096.239755] LNet: Accept secure, port 988 [ 4097.856187] Key type lgssc registered [ 4098.555706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4103.038452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.299458] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.334486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.715607] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4109.795745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.579964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.751568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.998585] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4118.476692] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4122.351438] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4126.340852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.516751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.991703] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4133.093210] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4134.705808] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 4143.015112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.018436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4143.021243] Lustre: Skipped 1 previous similar message [ 4143.027280] Lustre: Skipped 1 previous similar message [ 4148.202554] Lustre: server umount lustre-OST0000 complete [ 4152.800848] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4152.803957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.809305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4156.364963] Lustre: server umount lustre-MDT0000 complete [ 4157.892235] LustreError: 291159:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050774 with bad export cookie 5867996655627304047 [ 4157.895440] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.896606] LustreError: 291159:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4158.096820] Lustre: server umount lustre-MDT0001 complete [ 4163.710210] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4164.792816] Key type lgssc unregistered [ 4164.931062] LNet: 294832:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4164.934133] LNet: Removed LNI 192.168.201.131@tcp [ 4165.250757] Key type .llcrypt unregistered [ 4165.252852] Key type ._llcrypt unregistered [ 4175.660930] alg: No test for adler32 (adler32-zlib) [ 4176.413298] Key type ._llcrypt registered [ 4176.414391] Key type .llcrypt registered [ 4176.462503] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4176.938502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4177.100475] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4177.212176] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4177.214777] LNet: Accept secure, port 988 [ 4178.800093] Key type lgssc registered [ 4179.197120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4182.178143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.340323] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.362692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.120409] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4188.539217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.683840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4190.130912] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4192.960135] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4195.337965] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4197.915403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4198.031420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.762687] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4203.012119] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4213.414855] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 4223.973146] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4223.975828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.977121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4223.982641] Lustre: Skipped 1 previous similar message [ 4223.987839] Lustre: Skipped 1 previous similar message [ 4228.660417] Lustre: server umount lustre-OST0000 complete [ 4234.209688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.210399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.214143] Lustre: Skipped 1 previous similar message [ 4236.567374] Lustre: server umount lustre-MDT0000 complete [ 4238.348488] LustreError: 296272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050854 with bad export cookie 1103686167246344501 [ 4238.350847] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4238.355487] LustreError: 296272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4238.511137] Lustre: server umount lustre-MDT0001 complete [ 4243.917104] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4245.173691] Key type lgssc unregistered [ 4245.319897] LNet: 300784:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4245.324352] LNet: Removed LNI 192.168.201.131@tcp [ 4245.755078] Key type .llcrypt unregistered [ 4245.756531] Key type ._llcrypt unregistered [ 4254.681601] alg: No test for adler32 (adler32-zlib) [ 4255.435362] Key type ._llcrypt registered [ 4255.437224] Key type .llcrypt registered [ 4255.508341] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4256.100568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4256.308275] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4256.408728] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4256.414140] LNet: Accept secure, port 988 [ 4258.032069] Key type lgssc registered [ 4258.584944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4262.326883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.600968] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.626706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.843407] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4269.025620] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.173427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.338757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4271.995732] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4275.106327] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4277.364403] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4280.348808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4280.512801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.299916] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4285.615378] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4293.864053] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4301.281843] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4301.289066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.294365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4301.793893] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4301.796699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.801985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.353830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.356762] Lustre: Skipped 1 previous similar message [ 4307.051813] Lustre: server umount lustre-OST0000 complete [ 4309.474147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.475199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4309.480071] Lustre: Skipped 1 previous similar message [ 4314.593385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.598133] Lustre: Skipped 2 previous similar messages [ 4323.296556] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4323.345316] Lustre: server umount lustre-MDT0000 complete [ 4324.834316] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.842894] LustreError: Skipped 1 previous similar message [ 4325.506381] LustreError: 302031:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050941 with bad export cookie 14355487729328143835 [ 4325.510522] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.514175] LustreError: 302031:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4325.712059] Lustre: server umount lustre-MDT0001 complete [ 4331.662238] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4332.872457] Key type lgssc unregistered [ 4333.025665] LNet: 305909:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4333.029172] LNet: Removed LNI 192.168.201.131@tcp [ 4333.395799] Key type .llcrypt unregistered [ 4333.397064] Key type ._llcrypt unregistered [ 4341.404572] alg: No test for adler32 (adler32-zlib) [ 4342.157427] Key type ._llcrypt registered [ 4342.158982] Key type .llcrypt registered [ 4342.201205] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4342.561349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4342.818073] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4342.910686] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4342.913811] LNet: Accept secure, port 988 [ 4344.520113] Key type lgssc registered [ 4344.922087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4348.155462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.392841] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.417096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.049863] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4354.438800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.528891] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4354.617316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4356.530533] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4359.530665] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4361.945514] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4365.122904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.295892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.199403] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4370.711290] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4379.634096] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 4384.736918] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.739100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.740783] LustreError: Skipped 1 previous similar message [ 4384.741337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.751257] Lustre: Skipped 1 previous similar message [ 4386.466787] Lustre: server umount lustre-OST0000 complete [ 4389.857570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.861580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.865502] Lustre: Skipped 1 previous similar message [ 4389.867591] Lustre: Skipped 1 previous similar message [ 4394.376873] Lustre: server umount lustre-MDT0000 complete [ 4394.976882] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.981500] LustreError: Skipped 1 previous similar message [ 4395.813208] LustreError: 307155:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051012 with bad export cookie 6046811325357734294 [ 4395.816139] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.819024] LustreError: 307155:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4395.935174] Lustre: server umount lustre-MDT0001 complete [ 4400.684642] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4401.686244] Key type lgssc unregistered [ 4401.798333] LNet: 310844:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4401.801421] LNet: Removed LNI 192.168.201.131@tcp [ 4402.079277] Key type .llcrypt unregistered [ 4402.080410] Key type ._llcrypt unregistered [ 4406.792020] alg: No test for adler32 (adler32-zlib) [ 4407.543381] Key type ._llcrypt registered [ 4407.544938] Key type .llcrypt registered [ 4407.589931] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4414.451231] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4420.575243] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4420.909916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.154278] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4421.235282] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4421.237685] LNet: Accept secure, port 988 [ 4422.824133] Key type lgssc registered [ 4423.200722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4426.733394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4429.666819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4432.387066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.856756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4437.798505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4437.845187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.955958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4438.970801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4439.010180] Lustre: lustre-MDT0000: new disk, initializing [ 4439.045501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.054594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4440.790767] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4445.248520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4445.270488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.305705] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4445.323298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4445.325364] Lustre: Skipped 1 previous similar message [ 4445.357794] Lustre: lustre-MDT0001: new disk, initializing [ 4445.387170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4445.401582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4445.405584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4446.821401] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4450.631665] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4452.434493] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4454.793036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4454.818296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.821046] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4454.914841] Lustre: lustre-OST0000: new disk, initializing [ 4454.917201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4454.955548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.458068] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4460.543675] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4461.565353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4461.568617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4462.169967] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4462.259382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4463.831891] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4463.919868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4466.658767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.659531] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4466.666467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4466.670594] Lustre: Skipped 1 previous similar message [ 4470.762706] Lustre: server umount lustre-OST0000 complete [ 4476.384983] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4476.388184] LustreError: Skipped 1 previous similar message [ 4476.389818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.394220] Lustre: Skipped 1 previous similar message [ 4476.396318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4478.407447] Lustre: server umount lustre-MDT0000 complete [ 4479.913193] LustreError: 315440:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051096 with bad export cookie 3641791153779156057 [ 4479.915754] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.918929] LustreError: 315440:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4480.071914] Lustre: server umount lustre-MDT0001 complete [ 4483.728335] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:25:00 (1781051100) [ 4486.518026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.704286] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.731938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.288966] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4491.341308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.458480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.923746] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4495.648128] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4497.727707] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4500.306574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.423784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.968828] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4504.813923] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4512.736798] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4512.739525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.743234] Lustre: Skipped 1 previous similar message [ 4512.744836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4512.746393] Lustre: Skipped 1 previous similar message [ 4516.834189] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4516.834597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4516.837163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.502692] Lustre: server umount lustre-OST0000 complete [ 4521.953174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.957874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.960491] Lustre: Skipped 2 previous similar messages [ 4525.956287] Lustre: server umount lustre-MDT0000 complete [ 4527.072964] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.077561] LustreError: Skipped 1 previous similar message [ 4527.238984] LustreError: 321991:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051143 with bad export cookie 3641791153779156911 [ 4527.241783] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4527.244525] LustreError: 321991:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4531.992191] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4533.008331] Key type lgssc unregistered [ 4533.126335] LNet: 322793:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4533.129545] LNet: Removed LNI 192.168.201.131@tcp [ 4533.406227] Key type .llcrypt unregistered [ 4533.407171] Key type ._llcrypt unregistered [ 4543.214602] alg: No test for adler32 (adler32-zlib) [ 4543.966333] Key type ._llcrypt registered [ 4543.968140] Key type .llcrypt registered [ 4544.006809] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4549.395207] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4549.831956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4550.132864] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4550.222718] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4550.224485] LNet: Accept secure, port 988 [ 4551.816249] Key type lgssc registered [ 4552.238238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4555.466363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4557.744437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4559.740710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.706892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4563.279363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4565.064025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4567.542487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4567.567016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.684937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4568.696368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4568.737887] Lustre: lustre-MDT0000: new disk, initializing [ 4568.768308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.777577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4570.278667] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4574.760634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4574.781351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.817533] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4574.835513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4574.838672] Lustre: Skipped 1 previous similar message [ 4574.877961] Lustre: lustre-MDT0001: new disk, initializing [ 4574.913052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.925784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4574.929638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4576.436016] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4580.609562] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4582.661859] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4584.943969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4584.964822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.968621] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4585.053483] Lustre: lustre-OST2710: new disk, initializing [ 4585.055309] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 4585.082333] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4586.566972] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4590.586328] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4593.142726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4593.147619] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4593.200925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4593.227950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4593.231816] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4593.278979] Lustre: lustre-OST03e8: new disk, initializing [ 4593.280934] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 4593.311759] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4595.083743] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4598.003641] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4598.779314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4598.782619] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4601.055185] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 4601.138475] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4602.663738] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 4602.744319] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4604.278928] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 4604.349714] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4605.879928] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 4605.952963] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4608.994390] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.995207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4609.001038] Lustre: Skipped 3 previous similar messages [ 4609.003348] Lustre: Skipped 2 previous similar messages [ 4613.187922] Lustre: server umount lustre-MDT0000 complete [ 4614.112926] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.116986] LustreError: Skipped 3 previous similar messages [ 4614.294077] LustreError: 331810:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051230 with bad export cookie 6648351806710977274 [ 4614.296665] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.297713] LustreError: 331810:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4614.403801] Lustre: server umount lustre-MDT0001 complete [ 4620.768125] Lustre: 332212:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781051231/real 1781051231] req@00000000a838b048 x1867567507461824/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781051237 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4620.785635] Lustre: server umount lustre-OST2710 complete [ 4628.448093] Lustre: 332413:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781051238/real 1781051238] req@00000000a838b048 x1867567507462208/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781051244 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4628.518476] Lustre: server umount lustre-OST03e8 complete [ 4634.883693] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4638.230260] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4641.702258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4644.182218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.493541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4648.795822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4651.372681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4651.397841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.493679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4651.512585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4651.551377] Lustre: lustre-MDT0000: new disk, initializing [ 4651.586373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.595183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4653.324516] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4658.036767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.063452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.102239] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4658.203845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4659.656996] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4663.342717] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4665.281049] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4667.550856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.571427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.573519] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4667.651274] Lustre: lustre-OST0000: new disk, initializing [ 4667.653204] Lustre: Skipped 1 previous similar message [ 4667.656074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4667.659337] Lustre: Skipped 2 previous similar messages [ 4667.682672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.684221] Lustre: Skipped 1 previous similar message [ 4669.101544] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4670.392402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4670.395028] Lustre: Skipped 1 previous similar message [ 4670.396483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4673.248244] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4674.938649] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4675.018829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4676.509858] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4676.596491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4680.674735] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4680.675145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.675659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.675664] Lustre: Skipped 1 previous similar message [ 4680.677943] LustreError: Skipped 1 previous similar message [ 4680.683478] Lustre: Skipped 1 previous similar message [ 4683.439785] Lustre: server umount lustre-OST0000 complete [ 4685.793197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.794045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4685.797928] Lustre: Skipped 1 previous similar message [ 4685.800153] Lustre: Skipped 1 previous similar message [ 4690.913282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4690.916108] Lustre: Skipped 2 previous similar messages [ 4696.032869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4696.035271] Lustre: Skipped 1 previous similar message [ 4699.104111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4699.149802] Lustre: server umount lustre-MDT0000 complete [ 4700.708673] LustreError: 335311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051317 with bad export cookie 6648351806710978877 [ 4700.710169] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4700.712666] LustreError: 335311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4704.032756] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:28:40 (1781051320) [ 4704.596448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4705.214276] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:28:41 (1781051321) [ 4706.408847] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4709.207510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4710.548904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4711.884590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4713.219972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4713.600413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.101662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.230632] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4716.241745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4716.302231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.895751] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4721.117628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.166691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4722.823231] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4725.966771] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4728.307177] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4730.877440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4730.907156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4730.973184] Lustre: Denying initial registration attempt from nid 192.168.201.131@tcp, specified as failover [ 4730.977019] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4730.981459] LustreError: 343335:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 4730.985329] LustreError: 343335:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4730.987573] LustreError: 343335:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4731.037030] Lustre: server umount lustre-OST0000 complete [ 4731.038785] Lustre: Skipped 1 previous similar message [ 4731.040490] LustreError: 343335:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4733.409717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.410233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.415123] Lustre: Skipped 1 previous similar message [ 4733.420322] Lustre: Skipped 3 previous similar messages [ 4747.232380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4748.685893] LustreError: 341375:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051365 with bad export cookie 6648351806710979731 [ 4748.688133] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4748.691166] LustreError: 341375:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4754.254764] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4755.268242] Key type lgssc unregistered [ 4755.384512] LNet: 344411:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4755.387940] LNet: Removed LNI 192.168.201.131@tcp [ 4755.678953] Key type .llcrypt unregistered [ 4755.680408] Key type ._llcrypt unregistered [ 4760.537981] alg: No test for adler32 (adler32-zlib) [ 4761.290338] Key type ._llcrypt registered [ 4761.291832] Key type .llcrypt registered [ 4761.328726] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4761.698049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4761.896970] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4761.980406] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4761.982420] LNet: Accept secure, port 988 [ 4763.568130] Key type lgssc registered [ 4763.943569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4766.572185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4767.929872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4769.405136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4771.065754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4771.475057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.250885] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4780.072940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.160459] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4781.172869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4781.233254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.813271] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4785.864490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.916511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4785.930569] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4786.018194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4787.562759] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4790.436044] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4792.862114] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4795.938501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4795.974073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4796.066344] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4796.140176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.856320] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4800.963256] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4804.066096] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4804.066744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.068825] LustreError: Skipped 1 previous similar message [ 4804.074558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4804.078735] Lustre: Skipped 1 previous similar message [ 4807.954357] Lustre: server umount lustre-OST0000 complete [ 4811.744912] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4811.749070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.753584] Lustre: Skipped 1 previous similar message [ 4811.755883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.304888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.305529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.365643] Lustre: server umount lustre-MDT0000 complete [ 4816.758052] LustreError: 348081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051433 with bad export cookie 3163768050985655911 [ 4816.760278] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.761817] LustreError: 348081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4816.893241] Lustre: server umount lustre-MDT0001 complete [ 4821.125182] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4822.225824] Key type lgssc unregistered [ 4822.346375] LNet: 350976:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4822.349840] LNet: Removed LNI 192.168.201.131@tcp [ 4822.659932] Key type .llcrypt unregistered [ 4822.661809] Key type ._llcrypt unregistered [ 4832.487572] alg: No test for adler32 (adler32-zlib) [ 4833.239325] Key type ._llcrypt registered [ 4833.240885] Key type .llcrypt registered [ 4833.276162] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4833.715335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4833.851453] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 4833.930956] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 4833.933285] LNet: Accept secure, port 988 [ 4835.512087] Key type lgssc registered [ 4835.856854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4838.558659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.701367] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.716085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.187218] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4844.068779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.185430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4845.596127] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4847.963675] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4850.066767] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4853.013597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4853.194845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.749458] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4863.948162] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4870.114925] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4870.120733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4870.130793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4870.637577] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 4874.729039] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.738972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.742473] Lustre: Skipped 1 previous similar message [ 4874.750808] Lustre: Skipped 2 previous similar messages [ 4876.373191] Lustre: server umount lustre-MDT0000 complete [ 4879.188992] LustreError: 352417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051495 with bad export cookie 3280135668103169883 [ 4879.191862] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4879.194878] LustreError: 352417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4879.845603] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.847279] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.892552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4879.907096] Lustre: Skipped 1 previous similar message [ 4880.865551] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.962140] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.963037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4884.975630] Lustre: Skipped 1 previous similar message [ 4885.485737] Lustre: server umount lustre-MDT0001 complete [ 4891.376575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4902.455947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.691261] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0x2d856202e060c75b to 0x2d856202e060e338 [ 4912.720147] Lustre: MGC192.168.201.131@tcp: Connection restored to 192.168.201.131@tcp (at 0@lo) [ 4913.327262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.498093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.588443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 4918.883491] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4921.825337] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.848198] LustreError: Skipped 3 previous similar messages [ 4929.838467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.057993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4934.191991] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 4935.157548] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4936.242957] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4941.118198] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4942.460966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4947.484638] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4948.907615] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4954.596883] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4954.606785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.615458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4955.618303] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4959.663528] Lustre: server umount lustre-OST0000 complete [ 4965.858144] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4965.866555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.872368] Lustre: Skipped 1 previous similar message [ 4968.438460] Lustre: server umount lustre-MDT0000 complete [ 4971.265116] LustreError: 354386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051587 with bad export cookie 3280135668103177016 [ 4971.271286] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.271404] LustreError: 354386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4979.299575] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 4981.201769] Key type lgssc unregistered [ 4981.416829] LNet: 358930:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4981.421950] LNet: Removed LNI 192.168.201.131@tcp [ 4981.999260] Key type .llcrypt unregistered [ 4982.001140] Key type ._llcrypt unregistered [ 4999.977497] alg: No test for adler32 (adler32-zlib) [ 5000.732554] Key type ._llcrypt registered [ 5000.738177] Key type .llcrypt registered [ 5000.864035] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5001.828701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5002.051265] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 5002.256693] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5002.259973] LNet: Accept secure, port 988 [ 5003.904163] Key type lgssc registered [ 5004.729964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5010.542715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.894360] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.980802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5014.931277] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5017.057799] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.447979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.740985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.137916] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5031.002446] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5034.995769] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5038.050968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.055471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5038.062439] Lustre: Skipped 1 previous similar message [ 5038.075239] Lustre: Skipped 1 previous similar message [ 5043.170211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5043.177598] Lustre: Skipped 1 previous similar message [ 5043.291192] Lustre: server umount lustre-MDT0000 complete [ 5046.099313] LustreError: 360373:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051662 with bad export cookie 5685806919608376752 [ 5046.099681] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5046.107295] LustreError: 360373:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5046.332702] Lustre: server umount lustre-MDT0001 complete [ 5056.874924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.088784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5057.109675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5057.211165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.922498] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5067.996900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.117269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5068.197466] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5068.442339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.308796] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5078.497798] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5082.951960] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5088.322416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.484400] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5088.492401] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5088.498947] LustreError: 365076:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 5088.506854] LustreError: 365076:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 5088.512449] LustreError: 365076:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 5088.609175] Lustre: server umount lustre-OST0000 complete [ 5088.613965] LustreError: 365076:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5093.069363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5093.194181] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5093.315425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.934631] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5100.088818] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 5101.121225] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5106.436591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5106.490954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5106.494978] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5106.526874] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5106.577797] Lustre: lustre-OST0001: new disk, initializing [ 5106.581914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5106.629776] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5108.161658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5108.168222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5110.181801] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5116.248926] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5123.556803] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5123.565198] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.575680] LustreError: Skipped 1 previous similar message [ 5123.590202] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5123.597904] Lustre: Skipped 1 previous similar message [ 5128.674285] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5128.678202] Lustre: Skipped 2 previous similar messages [ 5133.280365] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5133.312459] Lustre: server umount lustre-OST0001 complete [ 5133.802917] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.814411] LustreError: Skipped 1 previous similar message [ 5138.920392] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.922452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.929408] LustreError: Skipped 1 previous similar message [ 5138.947227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5138.950474] Lustre: Skipped 1 previous similar message [ 5144.038797] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.050287] LustreError: Skipped 1 previous similar message [ 5149.154343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.164071] Lustre: Skipped 2 previous similar messages [ 5150.176284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5150.314326] Lustre: server umount lustre-OST0000 complete [ 5154.277595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.290833] Lustre: Skipped 1 previous similar message [ 5159.398040] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.409768] LustreError: Skipped 3 previous similar messages [ 5159.428906] Lustre: server umount lustre-MDT0000 complete [ 5161.954681] LustreError: 363080:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051778 with bad export cookie 5685806919608377305 [ 5161.961159] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.966840] LustreError: 363080:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5170.313917] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5172.216681] Key type lgssc unregistered [ 5172.415567] LNet: 368449:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5172.421499] LNet: Removed LNI 192.168.201.131@tcp [ 5173.029399] Key type .llcrypt unregistered [ 5173.031192] Key type ._llcrypt unregistered [ 5199.261099] alg: No test for adler32 (adler32-zlib) [ 5200.021544] Key type ._llcrypt registered [ 5200.022731] Key type .llcrypt registered [ 5200.126148] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5209.736743] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5210.397114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5210.755436] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 5210.930241] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5210.932516] LNet: Accept secure, port 988 [ 5212.552116] Key type lgssc registered [ 5213.271731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5219.313510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5223.882764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5227.404591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5231.265043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5235.147848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5235.187232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.343364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5236.359636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5236.408870] Lustre: lustre-MDT0000: new disk, initializing [ 5236.462254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5236.478752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5239.005941] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5245.123135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5245.168113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.223992] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5245.253150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5245.255474] Lustre: Skipped 1 previous similar message [ 5245.295708] Lustre: lustre-MDT0001: new disk, initializing [ 5245.343802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5245.371704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5245.379098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5247.885769] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5253.295562] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.928612] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5259.215848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5259.248044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.252668] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5259.406136] Lustre: lustre-OST0000: new disk, initializing [ 5259.409778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5259.457323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5261.761852] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5264.904210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5264.908283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5267.032795] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5269.789320] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5269.925510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5272.479560] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5272.637122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5275.111062] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.111576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.114104] LustreError: Skipped 1 previous similar message [ 5275.115270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.127069] Lustre: Skipped 1 previous similar message [ 5280.134816] Lustre: server umount lustre-OST0000 complete [ 5285.345660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.347285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.357916] Lustre: Skipped 1 previous similar message [ 5285.366705] Lustre: Skipped 1 previous similar message [ 5289.146453] Lustre: server umount lustre-MDT0000 complete [ 5290.471286] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.488996] LustreError: Skipped 1 previous similar message [ 5291.499476] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5291.517361] LustreError: Skipped 1 previous similar message [ 5292.307410] LustreError: 372536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781051908 with bad export cookie 7909611419344536609 [ 5292.310091] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5292.315076] LustreError: 372536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5292.506365] Lustre: server umount lustre-MDT0001 complete [ 5298.538934] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 20:38:34 (1781051914) [ 5302.476100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5305.791099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5321.486370] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5327.504906] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5335.603718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5340.773941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5344.709874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5349.346249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5354.768746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5354.817258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.000256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5355.025615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5355.099784] Lustre: lustre-MDT0000: new disk, initializing [ 5355.183061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5355.202250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5358.744490] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5368.153186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5368.210993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.266471] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5368.297446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5368.299620] Lustre: Skipped 1 previous similar message [ 5368.350929] Lustre: lustre-MDT0001: new disk, initializing [ 5368.416804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5368.452754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5368.470473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5373.125311] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5382.184299] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5386.752686] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5391.477740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.525625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5391.530235] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5391.777815] Lustre: lustre-OST0000: new disk, initializing [ 5391.781428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5391.853337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.451451] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5397.074651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5397.091611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5404.090829] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5408.655884] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5408.807259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5412.998931] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5413.204302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5417.955665] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5417.960138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.966470] LustreError: Skipped 1 previous similar message [ 5417.983566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5417.999636] Lustre: Skipped 1 previous similar message [ 5423.076967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5423.093478] Lustre: Skipped 1 previous similar message [ 5428.197968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.206328] Lustre: Skipped 1 previous similar message [ 5429.216223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5429.402847] Lustre: server umount lustre-OST0000 complete [ 5433.313463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.325195] Lustre: Skipped 1 previous similar message [ 5438.434879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5438.440816] Lustre: Skipped 3 previous similar messages [ 5438.966974] Lustre: server umount lustre-MDT0000 complete [ 5442.142989] LustreError: 381154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052058 with bad export cookie 7909611419344537449 [ 5442.152315] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5442.152051] LustreError: 381154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5449.316389] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 20:41:05 (1781052065) [ 5455.657420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.955506] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.015740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.894700] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5461.474831] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.957763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.314077] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5473.789869] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5477.483823] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5482.641762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5485.944092] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5491.857976] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5501.458777] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 5502.646379] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5503.538211] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 5504.724663] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 5506.164750] Lustre: Failing over lustre-MDT0000 [ 5506.314830] Lustre: server umount lustre-MDT0000 complete [ 5506.316805] Lustre: Skipped 1 previous similar message [ 5506.539593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.541899] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.546090] Lustre: Skipped 2 previous similar messages [ 5506.560162] LustreError: Skipped 1 previous similar message [ 5511.137764] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5516.293725] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.300915] LustreError: Skipped 5 previous similar messages [ 5517.794813] Lustre: 370766:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781052127/real 1781052127] req@00000000808ff350 x1867568200596736/t0(0) o400->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781052134 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5517.817994] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5523.287374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.947504] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0x6dc495add1678abf to 0x6dc495add1678ea1 [ 5523.978017] Lustre: MGC192.168.201.131@tcp: Connection restored to 192.168.201.131@tcp (at 0@lo) [ 5524.165481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5524.168737] Lustre: Skipped 2 previous similar messages [ 5524.208498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5527.099234] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5527.523345] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5529.587098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5529.598233] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5529.633604] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5534.310704] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5535.496309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5538.562115] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 5539.933098] Lustre: Failing over lustre-MDT0000 [ 5540.050521] LustreError: 389092:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000015ad92a0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5540.069422] Lustre: server umount lustre-MDT0000 complete [ 5543.930030] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.937626] LustreError: Skipped 14 previous similar messages [ 5544.929494] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5544.932620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.954885] Lustre: Skipped 3 previous similar messages [ 5552.098338] Lustre: 370767:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781052161/real 1781052161] req@00000000a7644e9a x1867568200605632/t0(0) o400->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781052168 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5552.118314] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5556.801335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.242420] Lustre: Evicted from MGS (at 192.168.201.131@tcp) after server handle changed from 0x6dc495add1678ea1 to 0x6dc495add16791b1 [ 5558.249948] Lustre: MGC192.168.201.131@tcp: Connection restored to (at 0@lo) [ 5558.255910] Lustre: Skipped 2 previous similar messages [ 5558.461876] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5561.944218] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5562.339381] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5563.888118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5563.889500] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5563.918154] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5568.920320] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5569.948816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5574.115737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.117363] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5574.123703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.125751] Lustre: Skipped 1 previous similar message [ 5580.050106] Lustre: server umount lustre-MDT0000 complete [ 5582.840264] LustreError: 384935:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052199 with bad export cookie 7909611419344540081 [ 5582.840701] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.848495] LustreError: 384935:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5592.032147] Lustre: 390740:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781052202/real 1781052202] req@0000000074d59b06 x1867568200615360/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781052208 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5601.876888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.171132] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.185157] LustreError: Skipped 19 previous similar messages [ 5602.217832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.225179] Lustre: Skipped 1 previous similar message [ 5605.076362] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5610.877275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.982435] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5619.723440] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5623.825540] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5628.290102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5631.197423] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5632.621125] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5637.117289] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5646.305204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.327789] Lustre: Skipped 1 previous similar message [ 5646.330208] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.333841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.336542] Lustre: Skipped 5 previous similar messages [ 5647.416460] Lustre: server umount lustre-OST0000 complete [ 5647.420099] Lustre: Skipped 2 previous similar messages [ 5658.836924] LustreError: 392354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052275 with bad export cookie 7909611419344540809 [ 5658.840031] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5658.852317] LustreError: 392354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5666.934920] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5668.599554] Key type lgssc unregistered [ 5668.820872] LNet: 395230:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5668.830837] LNet: Removed LNI 192.168.201.131@tcp [ 5669.336838] Key type .llcrypt unregistered [ 5669.338614] Key type ._llcrypt unregistered [ 5686.298234] alg: No test for adler32 (adler32-zlib) [ 5687.053328] Key type ._llcrypt registered [ 5687.055959] Key type .llcrypt registered [ 5687.153897] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5688.206276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5688.430342] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 5688.544019] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5688.547507] LNet: Accept secure, port 988 [ 5690.176098] Key type lgssc registered [ 5690.875700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5696.599588] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.605897] LustreError: 396716:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5696.612681] LustreError: 396716:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5696.616929] LustreError: 396716:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5696.621821] LustreError: 396716:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5696.634333] LustreError: 396716:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5701.665769] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5701.669559] LustreError: 397113:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5701.674260] LustreError: 397113:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5701.678917] LustreError: 397113:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5701.683127] LustreError: 397113:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5701.689065] LustreError: 397113:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5712.107662] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5713.858653] Key type lgssc unregistered [ 5714.107608] LNet: 398069:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5714.111842] LNet: Removed LNI 192.168.201.131@tcp [ 5714.719946] Key type .llcrypt unregistered [ 5714.724994] Key type ._llcrypt unregistered [ 5725.780842] alg: No test for adler32 (adler32-zlib) [ 5726.536422] Key type ._llcrypt registered [ 5726.537913] Key type .llcrypt registered [ 5726.615288] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5736.095370] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5736.834116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5737.116830] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 5737.271065] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5737.276530] LNet: Accept secure, port 988 [ 5738.929311] Key type lgssc registered [ 5739.664693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5745.142483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5748.881779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5752.715380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5757.004851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5761.139842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5761.203816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.358797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5762.377482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5762.440573] Lustre: lustre-MDT0000: new disk, initializing [ 5762.497955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.510790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5765.520254] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5773.008101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5773.050183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.136686] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5773.172588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5773.176325] Lustre: Skipped 1 previous similar message [ 5773.248934] Lustre: lustre-MDT0001: new disk, initializing [ 5773.321961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5773.352151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5773.356764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5776.154876] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5781.942475] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5785.547477] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5789.769869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5789.833812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.841281] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5789.979246] Lustre: lustre-OST0000: new disk, initializing [ 5789.984790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5790.035754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.258645] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5797.910348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5797.917135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5800.285932] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5803.550759] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5803.722514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5806.417517] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5806.553225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5808.099472] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.101796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.105215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.107080] LustreError: Skipped 1 previous similar message [ 5808.120526] Lustre: Skipped 1 previous similar message [ 5813.218411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.227342] Lustre: Skipped 2 previous similar messages [ 5813.868864] Lustre: server umount lustre-OST0000 complete [ 5818.337554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.346124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.349204] Lustre: Skipped 1 previous similar message [ 5823.458386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.464391] Lustre: Skipped 1 previous similar message [ 5828.578102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5828.582215] Lustre: Skipped 1 previous similar message [ 5830.112151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5830.266164] Lustre: server umount lustre-MDT0000 complete [ 5832.452633] LustreError: 402206:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052448 with bad export cookie 7758287684149039009 [ 5832.456685] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.458187] LustreError: 402206:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5832.715847] Lustre: server umount lustre-MDT0001 complete [ 5837.930817] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5838.907815] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:47:35 (1781052455) [ 5843.660644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.027749] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.064192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.014973] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5849.573087] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.643934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.832452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.668435] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5860.039720] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5863.102359] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5866.838304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.018741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.433533] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5874.765566] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5879.432944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5879.486712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.491227] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5879.572582] Lustre: lustre-OST0001: new disk, initializing [ 5879.575736] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5882.594949] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 5885.462475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5885.467707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5888.808726] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5895.656078] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.658124] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.670290] Lustre: Skipped 1 previous similar message [ 5895.675395] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5895.683732] Lustre: Skipped 2 previous similar messages [ 5905.888427] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5905.920446] Lustre: server umount lustre-OST0001 complete [ 5911.016714] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.033380] LustreError: Skipped 1 previous similar message [ 5916.131301] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.139248] LustreError: Skipped 1 previous similar message [ 5920.238932] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5920.246395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.256208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.259229] Lustre: Skipped 6 previous similar messages [ 5921.258611] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.277952] LustreError: Skipped 1 previous similar message [ 5931.491095] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.497188] LustreError: Skipped 3 previous similar messages [ 5934.048124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5934.155333] Lustre: server umount lustre-OST0000 complete [ 5936.613579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.621541] Lustre: Skipped 2 previous similar messages [ 5942.819606] Lustre: server umount lustre-MDT0000 complete [ 5945.423541] LustreError: 405925:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052561 with bad export cookie 7758287684149039863 [ 5945.435734] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.436926] LustreError: 405925:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5958.673749] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 5960.366454] Key type lgssc unregistered [ 5960.559755] LNet: 410844:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5960.570271] LNet: Removed LNI 192.168.201.131@tcp [ 5961.055892] Key type .llcrypt unregistered [ 5961.057497] Key type ._llcrypt unregistered [ 5980.114450] alg: No test for adler32 (adler32-zlib) [ 5980.866329] Key type ._llcrypt registered [ 5980.867619] Key type .llcrypt registered [ 5980.912421] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 5988.588532] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5989.234869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5989.409612] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 5989.519176] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 5989.522617] LNet: Accept secure, port 988 [ 5991.128632] Key type lgssc registered [ 5991.608987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5995.747571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5998.942416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.248055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6005.786168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.761649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.803590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.921800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6010.934985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6010.978083] Lustre: lustre-MDT0000: new disk, initializing [ 6011.012749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.022486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6012.863706] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6020.211278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.254397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.348792] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6020.382327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6020.385122] Lustre: Skipped 1 previous similar message [ 6020.437660] Lustre: lustre-MDT0001: new disk, initializing [ 6020.503853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6020.552907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6020.561882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6023.992481] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6029.532973] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6032.372639] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6036.099724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6036.134685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6036.138717] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6036.273456] Lustre: lustre-OST0000: new disk, initializing [ 6036.278412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6036.325075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.463764] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6040.054513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6040.061975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6043.557321] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6045.748519] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6045.846048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6048.023902] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6048.141577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6050.280536] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6050.280988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.288598] LustreError: Skipped 1 previous similar message [ 6050.289332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.301930] Lustre: Skipped 1 previous similar message [ 6055.361532] Lustre: server umount lustre-OST0000 complete [ 6060.515570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6060.517537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6060.521284] Lustre: Skipped 1 previous similar message [ 6060.531675] Lustre: Skipped 2 previous similar messages [ 6065.633876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6065.633876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.778200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.783300] Lustre: Skipped 3 previous similar messages [ 6072.288178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6072.386267] Lustre: server umount lustre-MDT0000 complete [ 6074.475738] LustreError: 414927:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052690 with bad export cookie 13144852376895491899 [ 6074.484136] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.654624] Lustre: server umount lustre-MDT0001 complete [ 6079.316740] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 20:51:35 (1781052695) [ 6083.660250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6087.556863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.739460] Lustre: lustre-MDT0000: new disk, initializing [ 6087.751034] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.780338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.803680] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6093.285136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.538560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.712058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.801446] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6100.565887] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6104.213970] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6108.130167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.131595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.134349] Lustre: Skipped 1 previous similar message [ 6111.364488] Lustre: server umount lustre-MDT0000 complete [ 6113.582137] LustreError: 421755:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052729 with bad export cookie 13144852376895492753 [ 6113.586352] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6113.589786] LustreError: 421755:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6119.403644] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:52:15 (1781052735) [ 6123.267567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.492109] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.541681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.660203] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6128.609175] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.460834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.319939] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6137.231845] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6140.347420] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6143.551307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6143.747986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.752319] Lustre: Skipped 1 previous similar message [ 6145.858989] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6149.861401] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6158.857709] Lustre: Permanently deactivating lustre-OST0000 [ 6158.866271] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6164.982741] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6165.654625] LustreError: 425719:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 6165.660543] LustreError: 425719:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 6165.664432] LustreError: 425719:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 6166.623663] LustreError: 425768:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 6166.629118] LustreError: 425768:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 6173.613832] Lustre: server umount lustre-OST0000 complete [ 6173.615890] Lustre: Skipped 1 previous similar message [ 6175.024913] LustreError: 426017:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 6175.027741] LustreError: 426017:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 6176.736848] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6176.740922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6176.746582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6176.748765] Lustre: Skipped 3 previous similar messages [ 6182.371245] Lustre: 426114:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781052792/real 1781052792] req@000000005875cb8d x1867569016380480/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781052798 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6182.472938] Lustre: server umount lustre-MDT0000 complete [ 6183.906904] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.918604] LustreError: Skipped 1 previous similar message [ 6184.490193] LustreError: 422656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052800 with bad export cookie 13144852376895493306 [ 6184.492344] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6184.498069] LustreError: 422656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6190.560166] Lustre: 426315:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781052800/real 1781052800] req@0000000082d22fb1 x1867569016381504/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781052806 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6194.680676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.072264] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6198.210247] LustreError: 427225:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 6201.027712] Lustre: 427570:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.131@tcp [ 6208.483717] Lustre: 427669:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781052818/real 1781052818] req@0000000078aac840 x1867569016381952/t0(0) o251->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781052824 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6208.504584] Lustre: server umount MGS complete [ 6208.506346] Lustre: Skipped 1 previous similar message [ 6214.832464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.095147] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.109231] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6215.113556] Lustre: Skipped 1 previous similar message [ 6215.155230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6217.477698] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6221.408638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.545404] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6223.390651] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6227.019872] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6230.003812] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6230.977953] Lustre: Permanently reactivating lustre-OST0000 [ 6230.990335] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6230.993233] Lustre: Skipped 2 previous similar messages [ 6249.356381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6251.149553] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6254.179987] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6269.923758] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.926952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.935618] Lustre: Skipped 1 previous similar message [ 6269.938847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.941635] Lustre: Skipped 3 previous similar messages [ 6272.359926] Lustre: server umount lustre-OST0000 complete [ 6288.354409] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6289.885043] LustreError: 429086:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052906 with bad export cookie 13144852376895494237 [ 6289.887540] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6289.890472] LustreError: 429086:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6295.672708] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6296.943676] Key type lgssc unregistered [ 6297.100540] LNet: 432661:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6297.107442] LNet: Removed LNI 192.168.201.131@tcp [ 6297.579821] Key type .llcrypt unregistered [ 6297.580916] Key type ._llcrypt unregistered [ 6305.856060] alg: No test for adler32 (adler32-zlib) [ 6306.608420] Key type ._llcrypt registered [ 6306.610110] Key type .llcrypt registered [ 6306.662522] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 6313.595551] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6314.006892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6314.165629] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6314.248948] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6314.250576] LNet: Accept secure, port 988 [ 6315.832762] Key type lgssc registered [ 6316.267202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6320.023600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6322.989884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6325.558754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6328.241099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6331.290157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6331.328152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.440077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6332.454224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6332.495351] Lustre: lustre-MDT0000: new disk, initializing [ 6332.539359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.547755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6334.745044] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6339.544853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.568464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.602958] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6339.627671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6339.630143] Lustre: Skipped 1 previous similar message [ 6339.666966] Lustre: lustre-MDT0001: new disk, initializing [ 6339.712788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.736936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6339.741518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6341.436422] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6345.588469] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6347.701552] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6350.076655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6350.114196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6350.117350] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6350.202647] Lustre: lustre-OST0000: new disk, initializing [ 6350.204565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6350.235181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.649719] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6353.849088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6353.852243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6355.419264] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6356.859566] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6356.939562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6358.447648] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6358.535145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6364.131527] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.131863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.135494] LustreError: Skipped 1 previous similar message [ 6364.141342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.144009] Lustre: Skipped 1 previous similar message [ 6365.405971] Lustre: server umount lustre-OST0000 complete [ 6369.250875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.251294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.258879] Lustre: Skipped 2 previous similar messages [ 6372.998124] Lustre: server umount lustre-MDT0000 complete [ 6374.328501] LustreError: 435925:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781052990 with bad export cookie 15009030962983346484 [ 6374.331450] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6374.332573] LustreError: 435925:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6374.453022] Lustre: server umount lustre-MDT0001 complete [ 6377.336419] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6377.938506] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:56:34 (1781052994) [ 6380.283930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.436961] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.457892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6381.938769] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6385.064231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.192607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.880934] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6389.810165] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6391.895486] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6394.375346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.475682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.828404] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6398.361173] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6399.045568] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6400.620897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6407.137960] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.141100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.145915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.148870] Lustre: Skipped 1 previous similar message [ 6411.233535] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6411.234584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.237478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.020711] Lustre: server umount lustre-OST0000 complete [ 6415.843354] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6415.846420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.850923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6415.853018] Lustre: Skipped 1 previous similar message [ 6420.613387] Lustre: server umount lustre-MDT0000 complete [ 6421.473783] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.478070] LustreError: Skipped 1 previous similar message [ 6421.924874] LustreError: 440516:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053038 with bad export cookie 15009030962983347345 [ 6421.927297] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.928632] LustreError: 440516:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6427.102358] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6428.141467] Key type lgssc unregistered [ 6428.254343] LNet: 444381:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6428.258784] LNet: Removed LNI 192.168.201.131@tcp [ 6428.566892] Key type .llcrypt unregistered [ 6428.568316] Key type ._llcrypt unregistered [ 6437.847542] alg: No test for adler32 (adler32-zlib) [ 6438.598410] Key type ._llcrypt registered [ 6438.600014] Key type .llcrypt registered [ 6438.633708] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6438.735329] Key type .llcrypt unregistered [ 6438.737084] Key type ._llcrypt unregistered [ 6445.883180] alg: No test for adler32 (adler32-zlib) [ 6446.634364] Key type ._llcrypt registered [ 6446.635441] Key type .llcrypt registered [ 6446.676290] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6447.068085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6447.220449] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6447.300416] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6447.302437] LNet: Accept secure, port 988 [ 6448.888060] Key type lgssc registered [ 6449.264131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6452.131117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.268147] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.283969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.789935] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6457.781871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.884843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.890732] LustreError: Skipped 1 previous similar message [ 6457.892301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.913273] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.917725] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 6458.918813] LustreError: Skipped 1 previous similar message [ 6459.325877] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6461.791945] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6463.970363] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.976099] LustreError: Skipped 1 previous similar message [ 6464.005204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6464.120476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6465.612252] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6474.208772] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.211097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.215376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.723182] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.723497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.732285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.940314] Lustre: server umount lustre-OST0000 complete [ 6483.938263] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6483.941735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.946918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.404149] Lustre: server umount lustre-MDT0000 complete [ 6487.762065] LustreError: 448459:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053104 with bad export cookie 7936612755111201962 [ 6487.764671] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6487.766659] LustreError: 448459:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6487.880733] Lustre: server umount lustre-MDT0001 complete [ 6492.553347] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6493.531909] Key type lgssc unregistered [ 6493.638435] LNet: 449970:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6493.641817] LNet: Removed LNI 192.168.201.131@tcp [ 6493.915710] Key type .llcrypt unregistered [ 6493.916694] Key type ._llcrypt unregistered [ 6502.922178] alg: No test for adler32 (adler32-zlib) [ 6503.674290] Key type ._llcrypt registered [ 6503.675517] Key type .llcrypt registered [ 6503.714247] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6504.142390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6504.278953] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6504.357189] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6504.359186] LNet: Accept secure, port 988 [ 6505.944096] Key type lgssc registered [ 6506.301181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6509.094486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.560282] LustreError: 451403:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 1: rc = -5 [ 6557.280154] LustreError: 451403:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 4: rc = -110 [ 6588.000176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.461995] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6592.078954] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6594.827221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.572064] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.608447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.641662] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 6611.333898] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6614.801273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.944976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6616.544050] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6619.310030] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6621.613887] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6625.250095] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.251130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.258352] LustreError: Skipped 1 previous similar message [ 6625.263891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6625.268170] Lustre: Skipped 1 previous similar message [ 6630.201276] Lustre: server umount lustre-OST0000 complete [ 6634.978553] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6634.981590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.985366] Lustre: Skipped 1 previous similar message [ 6634.988413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6637.777234] Lustre: server umount lustre-MDT0000 complete [ 6639.338348] LustreError: 454003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053255 with bad export cookie 14716095565516830126 [ 6639.340773] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.344881] LustreError: 454003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6639.479757] Lustre: server umount lustre-MDT0001 complete [ 6644.496091] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6645.707553] Key type lgssc unregistered [ 6645.821559] LNet: 455105:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6645.824511] LNet: Removed LNI 192.168.201.131@tcp [ 6646.123630] Key type .llcrypt unregistered [ 6646.125275] Key type ._llcrypt unregistered [ 6655.078400] alg: No test for adler32 (adler32-zlib) [ 6655.830393] Key type ._llcrypt registered [ 6655.832081] Key type .llcrypt registered [ 6655.870196] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6656.355197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6656.483684] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6656.559989] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6656.561874] LNet: Accept secure, port 988 [ 6658.144110] Key type lgssc registered [ 6658.499433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6661.205666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.359857] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.383562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.920839] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6667.119862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.261444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6668.819077] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6671.582568] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6673.482776] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6675.709679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6675.812395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.117952] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6677.861224] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 6679.494658] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6681.953757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.954681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.961266] Lustre: Skipped 2 previous similar messages [ 6681.964496] Lustre: Skipped 1 previous similar message [ 6686.183759] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.31@tcp (stopping) [ 6687.201659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6687.203243] Lustre: Skipped 2 previous similar messages [ 6687.494270] Lustre: server umount lustre-MDT0000 complete [ 6688.251955] LustreError: 456546:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.31@tcp arrived at 1781053304 with bad export cookie 8542233866562928249 [ 6688.257931] LustreError: 456546:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6688.930092] LustreError: 456546:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053305 with bad export cookie 8542233866562927402 [ 6688.933461] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.934466] LustreError: 456546:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6692.320900] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.321040] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.321128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.321663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6692.325683] LustreError: Skipped 1 previous similar message [ 6693.344914] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6693.348466] LustreError: Skipped 1 previous similar message [ 6695.061661] Lustre: server umount lustre-OST0000 complete [ 6695.393205] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.398831] LustreError: Skipped 1 previous similar message [ 6697.443228] Lustre: server umount lustre-MDT0001 complete [ 6701.844054] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6702.808416] Key type lgssc unregistered [ 6702.913309] LNet: 460365:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6702.916106] LNet: Removed LNI 192.168.201.131@tcp [ 6703.182073] Key type .llcrypt unregistered [ 6703.182970] Key type ._llcrypt unregistered [ 6711.951071] alg: No test for adler32 (adler32-zlib) [ 6712.702510] Key type ._llcrypt registered [ 6712.703722] Key type .llcrypt registered [ 6712.735638] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6713.002685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6713.244768] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6713.316868] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6713.318190] LNet: Accept secure, port 988 [ 6714.888098] Key type lgssc registered [ 6715.197578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6717.555661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.674672] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6718.688334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.959770] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6722.601826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.702283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6724.066315] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6726.437051] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6728.263643] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6730.512852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6730.613946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.050246] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6734.644401] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6734.694824] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 6739.425908] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6739.429880] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.435504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6739.940369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.947668] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6740.960051] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.31@tcp (stopping) [ 6742.651781] Lustre: server umount lustre-MDT0001 complete [ 6745.057415] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6745.057486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.058240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.058688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.176964] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.177418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.421206] Lustre: server umount lustre-OST0000 complete [ 6756.832149] Lustre: 464889:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781053367/real 1781053367] req@0000000031c506f2 x1867569775520448/t0(0) o251->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781053373 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6756.904659] Lustre: server umount lustre-MDT0000 complete [ 6762.866904] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6763.936162] Key type lgssc unregistered [ 6764.039861] LNet: 465644:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6764.042568] LNet: Removed LNI 192.168.201.131@tcp [ 6764.321141] Key type .llcrypt unregistered [ 6764.322937] Key type ._llcrypt unregistered [ 6774.445839] alg: No test for adler32 (adler32-zlib) [ 6775.196321] Key type ._llcrypt registered [ 6775.197764] Key type .llcrypt registered [ 6775.232064] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 6781.443843] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6781.857963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6782.033713] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6782.117763] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6782.119502] LNet: Accept secure, port 988 [ 6783.712127] Key type lgssc registered [ 6784.121304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6787.365672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.715743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.723978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6793.890053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6796.784869] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6799.708224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.951409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.974269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.054409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6803.065298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6803.103231] Lustre: lustre-MDT0000: new disk, initializing [ 6803.133128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.140611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6804.647629] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6808.210686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6810.632479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6810.663685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.670626] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6810.763481] Lustre: lustre-OST0000: new disk, initializing [ 6810.765756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6810.768116] Lustre: Skipped 1 previous similar message [ 6810.796492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.439375] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6816.939354] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6825.950920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.339482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.366979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.412427] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6828.436130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6828.478709] Lustre: lustre-MDT0001: new disk, initializing [ 6828.504931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6828.508989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6828.537424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.246123] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6834.062933] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 6834.138974] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6835.628223] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 6836.727588] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6840.802037] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.802425] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6840.802975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6840.806936] Lustre: Skipped 1 previous similar message [ 6843.367528] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.31@tcp (stopping) [ 6843.370460] Lustre: Skipped 1 previous similar message [ 6844.548559] Lustre: server umount lustre-MDT0001 complete [ 6845.922442] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.926859] LustreError: Skipped 1 previous similar message [ 6851.041991] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.047370] LustreError: Skipped 1 previous similar message [ 6856.161765] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.165748] LustreError: Skipped 1 previous similar message [ 6859.232795] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6859.235653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.239547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6861.281671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6861.281903] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.196120] Lustre: server umount lustre-OST0000 complete [ 6876.128204] Lustre: 474052:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781053486/real 1781053486] req@00000000035814d5 x1867569847877184/t0(0) o251->MGC192.168.201.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781053492 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6876.193489] Lustre: server umount lustre-MDT0000 complete [ 6881.794699] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6882.673785] Key type lgssc unregistered [ 6882.773387] LNet: 474806:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6882.775715] LNet: Removed LNI 192.168.201.131@tcp [ 6883.057673] Key type .llcrypt unregistered [ 6883.058684] Key type ._llcrypt unregistered [ 6897.256533] alg: No test for adler32 (adler32-zlib) [ 6898.008369] Key type ._llcrypt registered [ 6898.009497] Key type .llcrypt registered [ 6898.049490] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6898.142913] Key type .llcrypt unregistered [ 6898.143936] Key type ._llcrypt unregistered [ 6904.952810] alg: No test for adler32 (adler32-zlib) [ 6905.704352] Key type ._llcrypt registered [ 6905.705370] Key type .llcrypt registered [ 6905.741338] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6906.062839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6906.288996] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 6906.370209] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 6906.371977] LNet: Accept secure, port 988 [ 6907.952067] Key type lgssc registered [ 6908.289885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6910.881919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6913.171729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6915.229077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6917.517808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6917.539228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.619176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6918.628360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6918.665456] Lustre: lustre-MDT0000: new disk, initializing [ 6918.685833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.692212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6920.001132] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6923.816974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6923.840212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.873041] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6923.889252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6923.891085] Lustre: Skipped 1 previous similar message [ 6923.923716] Lustre: lustre-MDT0001: new disk, initializing [ 6923.951460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6923.964302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6923.967197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6925.258021] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6928.901397] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6930.710973] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6932.929116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6932.952412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.955104] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6933.040036] Lustre: lustre-OST0000: new disk, initializing [ 6933.042374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6933.067691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.366623] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6938.253893] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6942.195523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6942.199131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6947.298322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.299090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6947.302797] Lustre: Skipped 2 previous similar messages [ 6947.304843] Lustre: Skipped 1 previous similar message [ 6952.068324] Lustre: server umount lustre-MDT0000 complete [ 6952.419254] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.423839] LustreError: Skipped 2 previous similar messages [ 6953.257872] LustreError: 478288:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053569 with bad export cookie 4866389199700713920 [ 6953.259777] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6953.261322] LustreError: 478288:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6953.366397] Lustre: server umount lustre-MDT0001 complete [ 6960.608149] Lustre: 481547:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781053570/real 1781053570] req@00000000e3de4f01 x1867569977899008/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781053576 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6960.675885] Lustre: server umount lustre-OST0000 complete [ 6963.936177] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 21:06:20 (1781053580) [ 6964.258918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6966.300898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.431479] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.449499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.792837] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6970.692118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.804502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6972.185247] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6974.531790] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6976.351342] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6978.553164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6978.580567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.648435] Lustre: Found index 0 for lustre-OST0000, updating log [ 6979.958304] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 6981.797384] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 6982.400424] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6985.889550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6985.889962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6985.896510] Lustre: Skipped 1 previous similar message [ 6989.716109] Lustre: server umount lustre-OST0000 complete [ 6991.329195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.329372] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.333034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.336544] Lustre: Skipped 1 previous similar message [ 6996.448865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6996.450699] Lustre: Skipped 1 previous similar message [ 6998.596679] Lustre: server umount lustre-MDT0000 complete [ 6999.726964] LustreError: 482353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053616 with bad export cookie 4866389199700715243 [ 6999.729706] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6999.730332] LustreError: 482353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7002.469750] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 21:06:59 (1781053619) [ 7004.280384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7005.787917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.429548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7008.921157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7014.071547] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 7016.917482] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 7020.073734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7022.194348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7024.060282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7025.966619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7028.099914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7028.128600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.205120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7028.215589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7028.250774] Lustre: lustre-MDT0000: new disk, initializing [ 7028.277063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7028.279424] Lustre: Skipped 1 previous similar message [ 7028.285892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7029.681851] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 7033.763541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7033.786849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.815049] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7033.829096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7033.831665] Lustre: Skipped 1 previous similar message [ 7033.867731] Lustre: lustre-MDT0001: new disk, initializing [ 7033.903385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7033.907164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7035.234491] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 7038.915987] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7040.717642] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7042.884447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7042.907073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7042.909234] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7042.990633] Lustre: lustre-OST0000: new disk, initializing [ 7042.992704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7044.274772] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug -1 all 8 [ 7044.658578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7044.661381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7047.988938] Lustre: DEBUG MARKER: oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7049.383123] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7049.458365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7050.877404] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7050.950575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7054.818176] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7054.818294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.821460] LustreError: Skipped 1 previous similar message [ 7054.826836] Lustre: Skipped 1 previous similar message [ 7054.829681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7054.832277] Lustre: Skipped 1 previous similar message [ 7057.694341] Lustre: server umount lustre-OST0000 complete [ 7057.695786] Lustre: Skipped 1 previous similar message [ 7059.424823] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7059.427360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.057054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7065.059086] Lustre: Skipped 3 previous similar messages [ 7073.248097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7074.523188] LustreError: 489379:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781053690 with bad export cookie 4866389199700716286 [ 7074.526553] LustreError: 166-1: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7074.528494] LustreError: 489379:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7074.683875] Lustre: server umount lustre-MDT0001 complete [ 7074.685998] Lustre: Skipped 1 previous similar message [ 7082.192189] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 7084.958343] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 7088.424374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7091.030875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7093.029721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7095.169080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7095.686468] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6944 sec ========== 21:08:32 (1781053712) [ 7113.340124] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 7114.412172] Key type lgssc unregistered [ 7114.535397] LNet: 498108:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7114.537711] LNet: Removed LNI 192.168.201.131@tcp [ 7114.828140] Key type .llcrypt unregistered [ 7114.829547] Key type ._llcrypt unregistered