[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491383621 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009097] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011025] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.020763] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024675] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026034] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029089] *** VALIDATE tmpfs *** [ 0.030471] *** VALIDATE proc *** [ 0.031235] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040334] debug: unmapping init [mem 0xffffffff98a59000-0xffffffff98a60fff] [ 0.042220] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051283] rcu: Hierarchical SRCU implementation. [ 0.053469] smp: Bringing up secondary CPUs ... [ 0.054597] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.058317] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.139031] node 0 deferred pages initialised in 75ms [ 0.142558] devtmpfs: initialized [ 0.144267] x86/mm: Memory block size: 128MB [ 0.147829] gcov: version magic: 0x41383552 [ 0.150222] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152216] pinctrl core: initialized pinctrl subsystem [ 0.153176] [ 0.153450] ************************************************************* [ 0.154012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155012] ** ** [ 0.156009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157011] ** ** [ 0.158010] ** This means that this kernel is built to expose internal ** [ 0.159010] ** IOMMU data structures, which may compromise security on ** [ 0.160012] ** your system. ** [ 0.161009] ** ** [ 0.162011] ** If you see this message and you are not debugging the ** [ 0.163009] ** kernel, report this immediately to your vendor! ** [ 0.164012] ** ** [ 0.165012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166008] ************************************************************* [ 0.167515] NET: Registered protocol family 16 [ 0.168345] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171450] cpuidle: using governor menu [ 0.172000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174398] PCI: Using configuration type 1 for base access [ 0.176119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.185118] cryptd: max_cpu_qlen set to 1000 [ 0.188214] ACPI: Added _OSI(Module Device) [ 0.189010] ACPI: Added _OSI(Processor Device) [ 0.190009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194291] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197107] ACPI: Interpreter enabled [ 0.198041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199009] ACPI: Using IOAPIC for interrupt routing [ 0.200091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201336] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209335] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.212068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214173] acpiphp: Slot [2] registered [ 0.215099] acpiphp: Slot [5] registered [ 0.216031] acpiphp: Slot [6] registered [ 0.217019] acpiphp: Slot [7] registered [ 0.217994] acpiphp: Slot [8] registered [ 0.218071] acpiphp: Slot [9] registered [ 0.219095] acpiphp: Slot [10] registered [ 0.220050] acpiphp: Slot [3] registered [ 0.220977] acpiphp: Slot [4] registered [ 0.221062] acpiphp: Slot [11] registered [ 0.222014] acpiphp: Slot [12] registered [ 0.223068] acpiphp: Slot [13] registered [ 0.224078] acpiphp: Slot [14] registered [ 0.225066] acpiphp: Slot [15] registered [ 0.226050] acpiphp: Slot [16] registered [ 0.227065] acpiphp: Slot [17] registered [ 0.228056] acpiphp: Slot [18] registered [ 0.229043] acpiphp: Slot [19] registered [ 0.229992] acpiphp: Slot [20] registered [ 0.230092] acpiphp: Slot [21] registered [ 0.231066] acpiphp: Slot [22] registered [ 0.232107] acpiphp: Slot [23] registered [ 0.233070] acpiphp: Slot [24] registered [ 0.234087] acpiphp: Slot [25] registered [ 0.235079] acpiphp: Slot [26] registered [ 0.236084] acpiphp: Slot [27] registered [ 0.237099] acpiphp: Slot [28] registered [ 0.238067] acpiphp: Slot [29] registered [ 0.239043] acpiphp: Slot [30] registered [ 0.239998] acpiphp: Slot [31] registered [ 0.240945] PCI host bridge to bus 0000:00 [ 0.241016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.242017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.244024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.247032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.248023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.249171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.250841] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.252281] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257899] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.260037] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.261023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.263028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.264574] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.265671] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.266028] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.267707] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.269665] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.276015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.279825] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.282000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.290016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.298018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.314017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.324433] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.331016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.339014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.360014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.370321] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.378026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.385043] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.402027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.414274] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.424016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.432020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.454019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.463645] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.471022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.475028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.494024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.503089] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.510016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.518017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.532028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.547608] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.550340] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.552407] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.554482] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.556204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.560039] iommu: Default domain type: Passthrough [ 0.561401] SCSI subsystem initialized [ 0.563152] ACPI: bus type USB registered [ 0.564100] usbcore: registered new interface driver usbfs [ 0.566112] usbcore: registered new interface driver hub [ 0.568089] usbcore: registered new device driver usb [ 0.570289] pps_core: LinuxPPS API ver. 1 registered [ 0.572020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.576098] PTP clock support registered [ 0.578176] EDAC MC: Ver: 3.0.0 [ 0.581175] PCI: Using ACPI for IRQ routing [ 0.583086] NetLabel: Initializing [ 0.585018] NetLabel: domain hash size = 128 [ 0.586016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.589122] NetLabel: unlabeled traffic allowed by default [ 0.591183] vgaarb: loaded [ 0.593468] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.595021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.601005] clocksource: Switched to clocksource kvm-clock [ 0.713124] VFS: Disk quotas dquot_6.6.0 [ 0.714953] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.717438] *** VALIDATE ramfs *** [ 0.718611] *** VALIDATE hugetlbfs *** [ 0.720058] pnp: PnP ACPI init [ 0.722395] pnp: PnP ACPI: found 6 devices [ 0.748847] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.751222] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.753068] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.754430] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.755866] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.757317] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.760192] NET: Registered protocol family 2 [ 0.762995] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.768947] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.774302] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.779608] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.783635] TCP: Hash tables configured (established 65536 bind 65536) [ 0.786941] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.790830] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.794300] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.798070] NET: Registered protocol family 1 [ 0.800914] RPC: Registered named UNIX socket transport module. [ 0.802644] RPC: Registered udp transport module. [ 0.803882] RPC: Registered tcp transport module. [ 0.805570] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.807797] NET: Registered protocol family 44 [ 0.809497] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.811743] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.813775] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.816320] PCI: CLS 0 bytes, default 64 [ 0.817955] Unpacking initramfs... [ 2.222308] debug: unmapping init [mem 0xffff9dc33cc54000-0xffff9dc33ffbffff] [ 2.226068] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.227863] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.230311] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.727774] Initialise system trusted keyrings [ 2.729301] Key type blacklist registered [ 2.730898] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.739488] zbud: loaded [ 2.742381] *** VALIDATE nfs *** [ 2.743629] *** VALIDATE nfs4 *** [ 2.744991] pstore: using deflate compression [ 2.748694] Platform Keyring initialized [ 2.873429] NET: Registered protocol family 38 [ 2.875430] Key type asymmetric registered [ 2.877064] Asymmetric key parser 'x509' registered [ 2.879032] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.882309] io scheduler mq-deadline registered [ 2.883975] io scheduler kyber registered [ 2.885933] io scheduler bfq registered [ 2.888213] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.890627] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.893304] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.896093] ACPI: Power Button [PWRF] [ 2.901675] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.909215] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.926110] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.935357] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.955530] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.984130] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.012115] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.016955] Non-volatile memory driver v1.3 [ 3.019214] Linux agpgart interface v0.103 [ 3.061361] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.065644] vda: detected capacity change from 0 to 34816000 [ 3.081199] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.084176] vdb: detected capacity change from 0 to 1073741824 [ 3.100770] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.103356] vdc: detected capacity change from 0 to 2621440000 [ 3.120219] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.122826] vdd: detected capacity change from 0 to 2621440000 [ 3.136668] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.139023] vde: detected capacity change from 0 to 4294967296 [ 3.151710] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.154642] vdf: detected capacity change from 0 to 4294967296 [ 3.161506] libphy: Fixed MDIO Bus: probed [ 3.169818] usbcore: registered new interface driver usbserial_generic [ 3.172511] usbserial: USB Serial support registered for generic [ 3.175400] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.179669] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.181692] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.184043] mousedev: PS/2 mouse device common for all mice [ 3.187600] rtc_cmos 00:05: RTC can wake from S4 [ 3.189905] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.191177] rtc_cmos 00:05: registered as rtc0 [ 3.195786] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.196759] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.199097] intel_pstate: CPU model not supported [ 3.203467] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.208489] hid: raw HID events driver (C) Jiri Kosina [ 3.210750] usbcore: registered new interface driver usbhid [ 3.212676] usbhid: USB HID core driver [ 3.214237] drop_monitor: Initializing network drop monitor service [ 3.216468] Initializing XFRM netlink socket [ 3.218739] NET: Registered protocol family 10 [ 3.221814] Segment Routing with IPv6 [ 3.222936] NET: Registered protocol family 17 [ 3.224601] mpls_gso: MPLS GSO support [ 3.231904] RAS: Correctable Errors collector initialized. [ 3.233933] AVX version of gcm_enc/dec engaged. [ 3.235710] AES CTR mode by8 optimization enabled [ 3.312757] sched_clock: Marking stable (3312658296, 0)->(4249770212, -937111916) [ 3.316528] registered taskstats version 1 [ 3.318370] Loading compiled-in X.509 certificates [ 3.320937] zswap: loaded using pool lzo/zbud [ 3.345329] Key type big_key registered [ 3.358112] Key type encrypted registered [ 3.359855] ima: No TPM chip found, activating TPM-bypass! [ 3.362399] ima: Allocated hash algorithm: sha1 [ 3.363679] ima: No architecture policies found [ 3.364922] evm: Initialising EVM extended attributes: [ 3.366714] evm: security.selinux [ 3.368294] evm: security.ima [ 3.369608] evm: security.capability [ 3.371240] evm: HMAC attrs: 0x1 [ 3.373908] rtc_cmos 00:05: setting system clock to 2026-03-26 09:34:06 UTC (1774517646) [ 3.381213] debug: unmapping init [mem 0xffffffff99a03000-0xffffffff99bfffff] [ 3.384731] debug: unmapping init [mem 0xffffffff98782000-0xffffffff98a58fff] [ 3.395089] Write protecting the kernel read-only data: 28672k [ 3.398590] debug: unmapping init [mem 0xffffffff96e03000-0xffffffff96ffffff] [ 3.401113] debug: unmapping init [mem 0xffffffff97714000-0xffffffff977fffff] [ 3.468498] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.477352] systemd[1]: Detected virtualization kvm. [ 3.479358] systemd[1]: Detected architecture x86-64. [ 3.481229] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.509077] systemd[1]: No hostname configured. [ 3.510800] systemd[1]: Set hostname to . [ 3.512864] random: systemd: uninitialized urandom read (16 bytes read) [ 3.515413] systemd[1]: Initializing machine ID from random generator. [ 3.563403] random: ln: uninitialized urandom read (6 bytes read) [ 3.640688] random: systemd: uninitialized urandom read (16 bytes read) [ 3.643886] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.650477] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.655855] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.258966] device-mapper: uevent: version 1.0.3 [ 4.261117] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.626497] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.016264] virtio_net virtio0 ens2: renamed from eth0 [ 5.053699] scsi host0: ata_piix [ 5.086279] scsi host1: ata_piix [ 5.087908] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.090696] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.468388] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.878233] random: crng init done [ 9.879517] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.363598] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.520960] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.755874] SELinux: Disabled at runtime. [ 11.817590] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.825952] systemd[1]: Detected virtualization kvm. [ 11.827913] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.308627] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.313305] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.317754] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.322317] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.328399] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.335347] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.340343] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.446405] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Dev[ 12.816283] squashfs: version 4.0 (2009/01/31) Phillip Lougher ices. [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.074692] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.179404] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.325450] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.337143] EDAC sbridge: Ver: 1.1.2 [ 14.941910] Key type dns_resolver registered [ 15.254359] NFS: Registering the id_resolver key type [ 15.257354] Key type id_resolver registered [ 15.259175] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg315-server login: [ 37.601678] libcfs: loading out-of-tree module taints kernel. [ 37.620751] alg: No test for adler32 (adler32-zlib) [ 38.372530] Key type ._llcrypt registered [ 38.373795] Key type .llcrypt registered [ 38.429703] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 50.113718] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 51.686779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.131231] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 52.864127] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 52.867404] LNet: Accept secure, port 988 [ 54.591156] Key type lgssc registered [ 56.509417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.373315] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 78.816697] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.210654] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.746694] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.326865] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.078124] hrtimer: interrupt took 3114357 ns [ 119.152371] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 133.618811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.706709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.092719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 135.125477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 135.240666] Lustre: lustre-MDT0000: new disk, initializing [ 135.329440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.347225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.838959] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 152.650536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.744816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.863343] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 152.903651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 152.910273] Lustre: Skipped 1 previous similar message [ 153.066930] Lustre: lustre-MDT0001: new disk, initializing [ 153.126358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.173567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.201854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.000288] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 172.578325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.697461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.710503] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 172.965644] Lustre: lustre-OST0000: new disk, initializing [ 172.971715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 173.047553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.293400] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 178.200499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 178.204835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.804994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.892192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.902855] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 190.022640] Lustre: lustre-OST0001: new disk, initializing [ 190.030981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 190.121423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.021799] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 199.215287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.224841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.272782] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.111873] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 220.738576] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing check_logdir /tmp/testlogs/ [ 227.966630] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing yml_node [ 233.916817] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 237.092286] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 240.063720] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 242.090522] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Mar 26 05:38:03 EDT 2026 [ 252.419904] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 254.027600] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 270.817387] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 270.820499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.821454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.866017] Lustre: Skipped 2 previous similar messages [ 275.938466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.947586] Lustre: Skipped 6 previous similar messages [ 276.850425] Lustre: server umount lustre-MDT0000 complete [ 281.056931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.059828] LustreError: Skipped 3 previous similar messages [ 281.491796] LustreError: 9192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774517924 with bad export cookie 10541930337667608199 [ 281.498862] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 281.504577] LustreError: 9192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 286.176373] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.176985] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.186363] Lustre: Skipped 1 previous similar message [ 286.187677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 286.197461] LustreError: Skipped 1 previous similar message [ 286.207753] Lustre: Skipped 1 previous similar message [ 287.201094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.220338] LustreError: Skipped 1 previous similar message [ 287.920538] Lustre: server umount lustre-MDT0001 complete [ 292.386455] Lustre: server umount lustre-OST0000 complete [ 296.633582] Lustre: server umount lustre-OST0001 complete [ 303.754991] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 312.127916] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 321.529045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.736731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.868103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.135029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.765218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.845553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.029299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 351.053757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 351.108882] Lustre: lustre-MDT0000: new disk, initializing [ 351.163853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 351.178796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 355.723521] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 366.782081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.931505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.166469] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 367.209594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 367.217482] Lustre: Skipped 1 previous similar message [ 367.282532] Lustre: lustre-MDT0001: new disk, initializing [ 367.431624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 367.476910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 367.481774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 372.626531] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 385.132528] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 393.079992] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 402.004028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 402.105472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 402.112800] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 402.420135] Lustre: lustre-OST0000: new disk, initializing [ 402.424850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 402.506585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.137954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.148628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 407.194599] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 417.287928] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 422.288453] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 422.487910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 427.326176] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 427.557491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.047974] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.059725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.061047] LustreError: Skipped 1 previous similar message [ 430.061599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.061603] Lustre: Skipped 2 previous similar messages [ 430.123355] Lustre: Skipped 2 previous similar messages [ 435.176880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.195938] Lustre: Skipped 2 previous similar messages [ 435.659679] Lustre: server umount lustre-OST0000 complete [ 440.288533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.310277] Lustre: Skipped 1 previous similar message [ 445.408604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.424165] Lustre: Skipped 3 previous similar messages [ 445.562806] Lustre: server umount lustre-MDT0000 complete [ 449.469365] LustreError: 15166:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518092 with bad export cookie 10541930337667609655 [ 449.471401] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 449.474416] LustreError: 15166:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 454.418546] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:41:35 (1774518095) [ 457.895558] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 460.552445] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:41:41 (1774518101) [ 469.214696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.644922] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.708780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.653171] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 475.106892] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.158924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.395211] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 496.117047] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 501.740481] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 508.798120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 513.839862] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 521.480951] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 528.865541] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.879555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.896787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.640716] Lustre: server umount lustre-OST0000 complete [ 533.644830] Lustre: Skipped 1 previous similar message [ 540.136349] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.142964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.157404] Lustre: Skipped 1 previous similar message [ 546.290219] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.303192] LustreError: Skipped 1 previous similar message [ 546.917333] LustreError: 18846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518190 with bad export cookie 10541930337667610509 [ 546.924381] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.931342] LustreError: 18846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 558.024660] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 560.641963] Key type lgssc unregistered [ 560.859825] LNet: 22519:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.870416] LNet: Removed LNI 192.168.203.115@tcp [ 561.642269] Key type .llcrypt unregistered [ 561.650274] Key type ._llcrypt unregistered [ 582.723958] alg: No test for adler32 (adler32-zlib) [ 583.479535] Key type ._llcrypt registered [ 583.484829] Key type .llcrypt registered [ 583.603742] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 584.798230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 585.252444] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 585.433287] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 585.437636] LNet: Accept secure, port 988 [ 587.119139] Key type lgssc registered [ 588.141777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 595.054369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.502202] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.549552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.250366] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 601.571528] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.705852] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.876729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.332232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 614.712841] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 622.755567] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 629.592825] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 637.074175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.565196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.431467] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 646.766829] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 651.363944] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 661.482667] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 661.492573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.512785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 662.498074] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 662.515144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 662.520731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.935416] Lustre: server umount lustre-OST0000 complete [ 671.712464] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 671.720067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.737261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 671.749925] Lustre: Skipped 1 previous similar message [ 677.200779] Lustre: server umount lustre-MDT0000 complete [ 677.860752] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.876559] LustreError: Skipped 1 previous similar message [ 681.551340] LustreError: 23964:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518324 with bad export cookie 8278904132415998227 [ 681.554121] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.566368] LustreError: 23964:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 681.970882] Lustre: server umount lustre-MDT0001 complete [ 694.483162] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 697.801663] Key type lgssc unregistered [ 698.185570] LNet: 27643:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.196811] LNet: Removed LNI 192.168.203.115@tcp [ 699.352449] Key type .llcrypt unregistered [ 699.354858] Key type ._llcrypt unregistered [ 729.205030] alg: No test for adler32 (adler32-zlib) [ 729.972170] Key type ._llcrypt registered [ 729.973748] Key type .llcrypt registered [ 730.234634] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 732.196215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.054212] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 733.626427] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 733.638913] LNet: Accept secure, port 988 [ 735.455184] Key type lgssc registered [ 738.464664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 748.383219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.916953] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.019250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.884765] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 755.175718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.289769] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.502357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.766499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 767.058513] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 775.859580] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.347607] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 789.626520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 789.937319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.138401] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 800.204349] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 803.426873] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 810.466966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.470115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.474842] Lustre: Skipped 2 previous similar messages [ 810.481470] Lustre: Skipped 2 previous similar messages [ 811.432409] Lustre: server umount lustre-MDT0000 complete [ 815.589404] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.603122] LustreError: Skipped 2 previous similar messages [ 816.479588] LustreError: 29116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518459 with bad export cookie 7904111141529523260 [ 816.484805] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 816.488617] LustreError: 29116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 820.708198] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.708574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.732780] LustreError: Skipped 1 previous similar message [ 820.754982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 820.758936] Lustre: Skipped 1 previous similar message [ 822.188491] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 822.209861] Lustre: Skipped 2 previous similar messages [ 823.097905] Lustre: server umount lustre-MDT0001 complete [ 844.221018] Lustre: server umount lustre-OST0000 complete [ 859.684454] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 862.018407] Key type lgssc unregistered [ 862.269495] LNet: 33117:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.276894] LNet: Removed LNI 192.168.203.115@tcp [ 862.995799] Key type .llcrypt unregistered [ 862.998615] Key type ._llcrypt unregistered [ 881.878365] alg: No test for adler32 (adler32-zlib) [ 882.631633] Key type ._llcrypt registered [ 882.633743] Key type .llcrypt registered [ 882.754984] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 883.568843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 884.154304] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 884.315357] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 884.322339] LNet: Accept secure, port 988 [ 885.975206] Key type lgssc registered [ 886.952937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.164956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.594181] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.651156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.405890] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 901.095809] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.208748] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.620425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.883092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.100499] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 918.118575] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 922.794947] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 928.521098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.772591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.462586] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 938.276071] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 946.162857] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 954.194173] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 956.386253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.386830] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 956.392444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.406880] Lustre: Skipped 1 previous similar message [ 956.427236] LustreError: Skipped 1 previous similar message [ 959.299779] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 959.304623] Lustre: Skipped 1 previous similar message [ 959.718594] Lustre: server umount lustre-OST0000 complete [ 977.907685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.916131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.928789] Lustre: Skipped 1 previous similar message [ 977.946165] Lustre: Skipped 1 previous similar message [ 983.012068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.017467] Lustre: Skipped 1 previous similar message [ 983.151462] Lustre: server umount lustre-MDT0000 complete [ 986.552553] LustreError: 37627:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518629 with bad export cookie 14700449882177783216 [ 986.560326] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.568630] LustreError: 37627:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 993.043522] Lustre: server umount lustre-MDT0001 complete [ 1003.229751] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1005.642514] Key type lgssc unregistered [ 1005.845296] LNet: 38433:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.857304] LNet: Removed LNI 192.168.203.115@tcp [ 1006.658733] Key type .llcrypt unregistered [ 1006.660197] Key type ._llcrypt unregistered [ 1025.876825] alg: No test for adler32 (adler32-zlib) [ 1026.632965] Key type ._llcrypt registered [ 1026.635768] Key type .llcrypt registered [ 1026.745776] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1027.673907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.044971] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 1028.243460] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1028.245607] LNet: Accept secure, port 988 [ 1029.863448] Key type lgssc registered [ 1030.761181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.440628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.771613] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.831334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.844324] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1043.945982] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.058657] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.973708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.385320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.440378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1058.641517] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1062.667947] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1067.903107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.123499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.162594] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1071.562369] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1077.536461] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1086.074807] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1089.185208] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1095.660503] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1095.666440] LNetError: 39198:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1095.677046] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1100.583850] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-26 5h52m22s [ 1101.761530] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1105.241189] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1106.403064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.408604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.412991] Lustre: Skipped 1 previous similar message [ 1106.416219] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1106.422326] Lustre: Skipped 2 previous similar messages [ 1106.433302] LNetError: 39195:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1106.439734] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1109.627674] Lustre: server umount lustre-MDT0000 complete [ 1110.331217] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.525708] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1111.530490] LNetError: 39197:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1111.545312] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1111.807935] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1115.449304] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.460789] LustreError: Skipped 3 previous similar messages [ 1116.376701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.467207] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1116.467733] LustreError: 11-0: MGC192.168.203.115@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1116.469387] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1116.470695] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x39e68dac6fc6055a to 0x39e68dac6fc609ac [ 1116.476328] LNetError: 39196:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1116.476394] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1116.501504] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 1116.608791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.638827] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1118.811977] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1120.931668] Lustre: DEBUG MARKER: Wait for df (28897) ... [ 1121.736152] Lustre: DEBUG MARKER: done [ 1121.765220] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1121.773876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1121.777950] Lustre: Skipped 1 previous similar message [ 1126.884133] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.884345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.890669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.916050] Lustre: Skipped 2 previous similar messages [ 1130.119813] Lustre: server umount lustre-OST0000 complete [ 1137.120307] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.122686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.123832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.133450] Lustre: Skipped 1 previous similar message [ 1137.136414] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1137.145124] Lustre: Skipped 2 previous similar messages [ 1137.160224] LNetError: 39198:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1137.166916] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1140.716440] Lustre: server umount lustre-MDT0000 complete [ 1142.244048] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.250853] LustreError: Skipped 4 previous similar messages [ 1142.989617] LustreError: 39876:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518786 with bad export cookie 4172177876554484140 [ 1142.993310] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.995429] LustreError: 39876:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.165650] Lustre: server umount lustre-MDT0001 complete [ 1151.089829] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1153.065264] Key type lgssc unregistered [ 1153.314939] LNet: 44991:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.321646] LNet: Removed LNI 192.168.203.115@tcp [ 1153.890519] Key type .llcrypt unregistered [ 1153.892614] Key type ._llcrypt unregistered [ 1176.505152] alg: No test for adler32 (adler32-zlib) [ 1177.257465] Key type ._llcrypt registered [ 1177.259565] Key type .llcrypt registered [ 1177.340975] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1187.537377] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1188.391314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1188.639607] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 1188.816709] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1188.826903] LNet: Accept secure, port 988 [ 1190.479128] Key type lgssc registered [ 1191.300433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1198.196050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1203.012775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1207.143270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1211.388968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1215.871186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1215.931788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.139957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1217.177996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1217.244731] Lustre: lustre-MDT0000: new disk, initializing [ 1217.340279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.356176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1220.506544] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1228.945974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1229.027186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.108668] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1229.148980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1229.152408] Lustre: Skipped 1 previous similar message [ 1229.242736] Lustre: lustre-MDT0001: new disk, initializing [ 1229.328172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.364881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1229.384439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1232.734926] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1239.193269] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1242.803751] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1246.878705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1246.927374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.931219] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1247.089376] Lustre: lustre-OST0000: new disk, initializing [ 1247.093088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1247.157186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.028852] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1256.003172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1256.018464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1257.798695] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1261.735306] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1261.893427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1265.802734] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1265.973383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1271.777918] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1271.778813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.785961] LustreError: Skipped 1 previous similar message [ 1271.787064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1271.799449] Lustre: Skipped 1 previous similar message [ 1273.604431] Lustre: server umount lustre-OST0000 complete [ 1276.897275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.909141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1276.917251] Lustre: Skipped 2 previous similar messages [ 1282.017176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1282.020971] Lustre: Skipped 1 previous similar message [ 1282.672449] Lustre: server umount lustre-MDT0000 complete [ 1285.333384] LustreError: 49080:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774518928 with bad export cookie 17463871730593961242 [ 1285.336796] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.340635] LustreError: 49080:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1285.593481] Lustre: server umount lustre-MDT0001 complete [ 1291.311479] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 05:55:33 (1774518933) [ 1296.563745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.976539] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.042599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.080672] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1302.496495] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.510570] LustreError: Skipped 1 previous similar message [ 1306.583832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.835055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1310.720204] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1317.282835] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1321.326296] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1325.752958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.006383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.804378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1334.107911] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1342.855789] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-26 5h56m24s [ 1343.817980] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1348.702886] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1350.641144] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1350.694152] LustreError: 53629:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1351.142032] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1351.145326] LNetError: 47298:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1351.149445] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1370.977577] Lustre: lustre-MDT0000: Client 1512c500-a42f-4204-8702-37e52420916a (at 192.168.203.15@tcp) reconnecting [ 1390.791130] LustreError: 53629:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1392.458583] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1394.252527] Lustre: DEBUG MARKER: done [ 1398.753043] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1398.754774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.761858] LustreError: Skipped 1 previous similar message [ 1398.763852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1398.774157] Lustre: Skipped 2 previous similar messages [ 1398.791482] Lustre: Skipped 1 previous similar message [ 1403.878672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.885158] Lustre: Skipped 1 previous similar message [ 1404.307266] Lustre: server umount lustre-OST0000 complete [ 1408.995252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.017361] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1409.029217] LNetError: 47295:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1409.040739] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1413.024319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.030688] Lustre: Skipped 4 previous similar messages [ 1413.218405] Lustre: server umount lustre-MDT0000 complete [ 1415.940163] LustreError: 53616:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519059 with bad export cookie 17463871730593962096 [ 1415.947145] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.950011] LustreError: 53616:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1424.252842] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1425.911903] Key type lgssc unregistered [ 1426.096593] LNet: 58047:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.104666] LNet: Removed LNI 192.168.203.115@tcp [ 1426.611554] Key type .llcrypt unregistered [ 1426.614785] Key type ._llcrypt unregistered [ 1448.406687] alg: No test for adler32 (adler32-zlib) [ 1449.162410] Key type ._llcrypt registered [ 1449.164809] Key type .llcrypt registered [ 1449.270357] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1459.373713] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1460.146507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1460.403431] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 1460.544508] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1460.546332] LNet: Accept secure, port 988 [ 1462.167133] Key type lgssc registered [ 1462.859028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1468.046741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.058749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1476.174265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1480.232207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.308477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.342854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.461614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1485.479460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1485.535483] Lustre: lustre-MDT0000: new disk, initializing [ 1485.583040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.595592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1488.098499] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1494.979211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1495.024483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.086900] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1495.118971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1495.123429] Lustre: Skipped 1 previous similar message [ 1495.188932] Lustre: lustre-MDT0001: new disk, initializing [ 1495.247632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1495.274237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1495.286771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1497.603265] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1503.920460] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1507.201776] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1510.950828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.988740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1510.991061] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1511.138149] Lustre: lustre-OST0000: new disk, initializing [ 1511.144173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1511.188225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.468695] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1517.075060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1517.080692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1519.491459] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1522.095291] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1522.214976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1524.600522] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1524.722450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1527.264611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.270920] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1527.271308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1527.279172] Lustre: Skipped 1 previous similar message [ 1531.844464] Lustre: server umount lustre-OST0000 complete [ 1536.481284] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.485519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.502696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1536.504695] Lustre: Skipped 1 previous similar message [ 1537.504550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1537.514228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.699958] Lustre: server umount lustre-MDT0000 complete [ 1542.625253] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.632267] LustreError: Skipped 1 previous similar message [ 1542.802603] LustreError: 62990:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519185 with bad export cookie 15850465840132467075 [ 1542.804251] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1542.808036] LustreError: 62990:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1542.960994] Lustre: server umount lustre-MDT0001 complete [ 1547.337879] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 05:59:49 (1774519189) [ 1548.291768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1549.463682] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 05:59:51 (1774519191) [ 1556.900957] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1559.273652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.566930] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.601594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.165600] Lustre: Failing over lustre-MDT0000 [ 1560.169641] LustreError: 67150:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1560.176202] LustreError: 67150:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1560.181509] LustreError: 67150:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1566.687165] Lustre: 67201:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519203/real 1774519203] req@000000003f41cd4a x1860718142966272/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774519209 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1566.826391] Lustre: server umount lustre-MDT0000 complete [ 1571.437134] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:00:13 (1774519213) [ 1575.491029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.779447] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.829844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.792849] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1581.024210] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.050717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.280668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.272695] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1592.217466] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1595.802763] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1599.843189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1602.802856] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1607.280898] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.086204] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1624.547020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.553283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.558938] Lustre: Skipped 2 previous similar messages [ 1624.565537] Lustre: Skipped 3 previous similar messages [ 1629.667090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.670860] Lustre: Skipped 2 previous similar messages [ 1630.000781] Lustre: server umount lustre-MDT0000 complete [ 1632.347634] LustreError: 70551:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519275 with bad export cookie 15850465840132468202 [ 1632.348912] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1632.356945] LustreError: 70551:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1634.436738] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1637.196482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1641.823093] Lustre: 60365:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519277/real 1774519277] req@00000000d47abccc x1860718142983488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774519284 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1641.832839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.364296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.999136] Lustre: 60363:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519280/real 1774519280] req@00000000fd01d2b5 x1860718142983552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774519287 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1646.047333] Lustre: 60364:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519282/real 1774519282] req@000000007156fc6a x1860718142983744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774519289 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1646.070945] Lustre: 60364:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1648.993586] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0xdbf82ab3c58f2dea to 0xdbf82ab3c58f3576 [ 1649.005422] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 1649.203669] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.251899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.255759] Lustre: Skipped 1 previous similar message [ 1650.280906] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1651.980769] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1657.598352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.845144] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1658.862716] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1660.638622] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1665.168672] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1668.189585] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1673.183985] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1673.187628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.193571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1677.470660] Lustre: server umount lustre-MDT0000 complete [ 1677.473094] Lustre: Skipped 1 previous similar message [ 1679.329272] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.347237] LustreError: Skipped 5 previous similar messages [ 1679.817945] LustreError: 67750:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519322 with bad export cookie 15850465840132470134 [ 1679.818654] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.827720] LustreError: 67750:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1683.729529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1689.178527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.551741] Lustre: 60365:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519327/real 1774519327] req@0000000090ff5995 x1860718142997184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774519334 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1691.563838] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.570016] Lustre: Skipped 2 previous similar messages [ 1696.673111] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0xdbf82ab3c58f3576 to 0xdbf82ab3c58f3b81 [ 1696.687819] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [ 1696.821857] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.869820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.873415] Lustre: Skipped 1 previous similar message [ 1696.916757] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1699.186290] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1703.812326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.055483] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1706.463298] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1709.027733] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1709.030716] Lustre: Skipped 1 previous similar message [ 1710.830491] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.321508] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1719.263902] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1719.268166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1719.276535] Lustre: Skipped 5 previous similar messages [ 1724.019719] Lustre: server umount lustre-MDT0000 complete [ 1724.022423] Lustre: Skipped 1 previous similar message [ 1725.877792] LustreError: 67748:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519369 with bad export cookie 15850465840132471681 [ 1725.882398] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.885762] LustreError: 67748:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1729.271464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1731.362171] Lustre: DEBUG MARKER: files compared the same [ 1738.719611] Lustre: 76929:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519375/real 1774519375] req@00000000c1021186 x1860718143008448/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774519381 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1738.729833] Lustre: 76929:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1748.882558] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1750.467832] Key type lgssc unregistered [ 1750.631985] LNet: 77836:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.638438] LNet: Removed LNI 192.168.203.115@tcp [ 1751.078514] Key type .llcrypt unregistered [ 1751.082356] Key type ._llcrypt unregistered [ 1765.798409] alg: No test for adler32 (adler32-zlib) [ 1766.548849] Key type ._llcrypt registered [ 1766.550614] Key type .llcrypt registered [ 1766.616777] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1767.328852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1767.577655] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 1767.745160] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1767.753067] LNet: Accept secure, port 988 [ 1769.383108] Key type lgssc registered [ 1770.062726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1774.844942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.061470] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.100475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.537380] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all 8 [ 1781.217186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1783.651714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.822791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1786.456898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all 8 [ 1790.781951] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1794.280964] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1798.043668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1798.191300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.204470] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1800.637473] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all 8 [ 1804.627252] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1806.380291] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1814.496305] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1814.501544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.514033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1816.544043] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1816.544400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.544843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1819.515538] Lustre: server umount lustre-OST0000 complete [ 1821.664791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.673090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.676875] Lustre: Skipped 2 previous similar messages [ 1826.784825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1826.793163] Lustre: Skipped 1 previous similar message [ 1827.483030] Lustre: server umount lustre-MDT0000 complete [ 1829.687623] LustreError: 79280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519472 with bad export cookie 2987262079818792178 [ 1829.688247] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.692935] LustreError: 79280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1829.833851] Lustre: server umount lustre-MDT0001 complete [ 1836.804152] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1838.187284] Key type lgssc unregistered [ 1838.332869] LNet: 82951:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.335617] LNet: Removed LNI 192.168.203.115@tcp [ 1838.710759] Key type .llcrypt unregistered [ 1838.711915] Key type ._llcrypt unregistered [ 1850.441145] alg: No test for adler32 (adler32-zlib) [ 1851.192403] Key type ._llcrypt registered [ 1851.193469] Key type .llcrypt registered [ 1851.244529] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1851.697310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.890677] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 1852.027021] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1852.029708] LNet: Accept secure, port 988 [ 1853.655268] Key type lgssc registered [ 1854.106947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.886261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.423376] LustreError: 84385:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 1: rc = -5 [ 1906.143138] LustreError: 84385:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 1936.863159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.073394] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1943.130922] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1947.243492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.299784] Lustre: *** cfs_fail_loc=706, val=0*** [ 1958.516108] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.554421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.583586] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1960.561630] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1964.001231] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.332598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.499664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.526444] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1966.324324] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1969.894553] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1972.940476] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1974.245201] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.248773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.257459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.754815] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.760863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.769938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.876208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.885478] Lustre: Skipped 1 previous similar message [ 1980.156604] Lustre: server umount lustre-OST0000 complete [ 1984.992768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.993067] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1985.002465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1985.006520] Lustre: Skipped 1 previous similar message [ 1988.294084] Lustre: server umount lustre-MDT0000 complete [ 1990.060337] LustreError: 87317:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519633 with bad export cookie 4955251056816956535 [ 1990.063196] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.064632] LustreError: 87317:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1990.206506] Lustre: server umount lustre-MDT0001 complete [ 1995.919669] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1997.394702] Key type lgssc unregistered [ 1997.559553] LNet: 88119:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.564654] LNet: Removed LNI 192.168.203.115@tcp [ 1998.009417] Key type .llcrypt unregistered [ 1998.010964] Key type ._llcrypt unregistered [ 2010.062612] alg: No test for adler32 (adler32-zlib) [ 2010.815365] Key type ._llcrypt registered [ 2010.816782] Key type .llcrypt registered [ 2010.859660] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2011.271961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.485717] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2011.595048] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2011.597671] LNet: Accept secure, port 988 [ 2013.191113] Key type lgssc registered [ 2013.643289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.973279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.023724] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2023.361563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.600206] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.662918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2025.559146] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2029.025676] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.035257] LustreError: Skipped 2 previous similar messages [ 2030.085191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2030.260733] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.269312] LustreError: Skipped 1 previous similar message [ 2030.285056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.397208] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2035.680781] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.685932] LustreError: Skipped 2 previous similar messages [ 2036.024969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.056869] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2037.935982] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2039.080205] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2049.507676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.507799] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2049.517415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.529384] Lustre: Skipped 1 previous similar message [ 2053.042406] Lustre: server umount lustre-OST0000 complete [ 2059.749563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.751633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.752492] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2059.756995] Lustre: Skipped 2 previous similar messages [ 2059.770676] Lustre: Skipped 1 previous similar message [ 2060.935694] Lustre: server umount lustre-MDT0000 complete [ 2062.868396] Lustre: server umount lustre-MDT0001 complete [ 2069.983142] Lustre: 92982:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519707/real 1774519707] req@00000000114cd580 x1860718720724672/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774519713 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2070.007615] Lustre: server umount MGS complete [ 2076.183495] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:08:38 (1774519718) [ 2092.288183] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2097.027115] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2102.266154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2105.834630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2108.968117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2112.197520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2115.468814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2115.491879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.702459] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2123.058501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.089848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.214403] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2123.254843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2123.305845] Lustre: lustre-MDT0001: new disk, initializing [ 2123.355434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.374013] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2123.380278] LustreError: 97678:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2125.954741] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2132.562537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2132.605332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2132.611260] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2132.757457] Lustre: lustre-OST0000: new disk, initializing [ 2132.760478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2132.809875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.541098] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2141.505156] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.243040] Lustre: 99126:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@000000003a8d6047 x1860718720729280/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1774519797 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2145.698863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2145.720555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2145.764425] Lustre: lustre-MDT0000: new disk, initializing [ 2145.854416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.871632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2146.919536] Lustre: 99530:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000085fd18a2 x1860718720732288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1774519802 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2148.042344] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2151.327817] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2151.974520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2151.977847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2156.521646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2156.527980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2176.483668] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.490602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.501561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.585888] Lustre: server umount lustre-OST0000 complete [ 2182.112907] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.113645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.116199] LustreError: Skipped 1 previous similar message [ 2182.121860] Lustre: Skipped 1 previous similar message [ 2182.127504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.131823] Lustre: Skipped 2 previous similar messages [ 2187.232049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.234780] Lustre: Skipped 1 previous similar message [ 2187.879748] Lustre: server umount lustre-MDT0000 complete [ 2204.639129] Lustre: 100721:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774519841/real 1774519841] req@0000000066dd60b5 x1860718720742080/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774519847 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2204.662798] Lustre: server umount MGS complete [ 2204.663984] Lustre: Skipped 1 previous similar message [ 2211.254571] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:10:53 (1774519853) [ 2222.154905] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2223.794481] Key type lgssc unregistered [ 2223.991436] LNet: 102127:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.996782] LNet: Removed LNI 192.168.203.115@tcp [ 2224.467891] Key type .llcrypt unregistered [ 2224.469335] Key type ._llcrypt unregistered [ 2233.513314] alg: No test for adler32 (adler32-zlib) [ 2234.265366] Key type ._llcrypt registered [ 2234.267062] Key type .llcrypt registered [ 2234.336246] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2234.843989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.125770] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2235.261298] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2235.265485] LNet: Accept secure, port 988 [ 2236.863113] Key type lgssc registered [ 2237.374834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.421789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.646917] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.667385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.553827] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2248.161022] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.579751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.758633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2250.637891] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2254.266363] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2257.008774] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2260.228974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.440795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.544589] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2266.761852] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2273.771828] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2284.000980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.005683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.011450] Lustre: Skipped 1 previous similar message [ 2284.014821] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2287.603918] Lustre: server umount lustre-OST0000 complete [ 2294.240982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.243596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.251519] Lustre: Skipped 1 previous similar message [ 2294.258193] Lustre: Skipped 1 previous similar message [ 2295.968888] Lustre: server umount lustre-MDT0000 complete [ 2297.958142] LustreError: 103374:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519941 with bad export cookie 15042378852493162923 [ 2297.960694] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.961952] LustreError: 103374:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2298.113876] Lustre: server umount lustre-MDT0001 complete [ 2304.026218] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2305.330121] Key type lgssc unregistered [ 2305.459502] LNet: 107068:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.465041] LNet: Removed LNI 192.168.203.115@tcp [ 2305.856555] Key type .llcrypt unregistered [ 2305.857813] Key type ._llcrypt unregistered [ 2313.750533] alg: No test for adler32 (adler32-zlib) [ 2314.502336] Key type ._llcrypt registered [ 2314.504439] Key type .llcrypt registered [ 2314.558529] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2315.028166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.395919] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2315.504052] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2315.506831] LNet: Accept secure, port 988 [ 2317.095122] Key type lgssc registered [ 2317.581126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2320.650340] LustreError: 108194:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2320.652551] LustreError: 108195:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2320.658630] LustreError: 108194:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4996 [ 2320.662114] LustreError: 108194:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2320.670616] LustreError: 108194:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2320.679293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.983699] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.025077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.243497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.395344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.573514] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2331.033311] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing lsmod [ 2332.384051] LustreError: 109495:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2332.400687] LustreError: 109503:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2332.405046] LustreError: 109495:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4986 [ 2332.408644] LustreError: 109495:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2332.415132] LustreError: 109495:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2332.420509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.636564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.736056] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2336.739292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.745223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.760870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.763958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.773370] Lustre: Skipped 1 previous similar message [ 2337.780521] Lustre: Skipped 2 previous similar messages [ 2340.104789] Lustre: server umount lustre-MDT0000 complete [ 2341.827947] LustreError: 108211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774519984 with bad export cookie 14301508829259399950 [ 2341.832043] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.834197] LustreError: 108211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2341.848813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.848978] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.859103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2341.979332] Lustre: server umount lustre-MDT0001 complete [ 2343.749386] Lustre: server umount lustre-OST0000 complete [ 2347.597099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.856597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.893286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.887735] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2353.120965] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.725353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.913161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2356.055633] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2359.784917] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2362.498526] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2366.005033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2366.186662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.251113] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2372.536285] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2378.472479] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2388.962493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.963658] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2388.964253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2388.970608] Lustre: Skipped 1 previous similar message [ 2391.282842] Lustre: server umount lustre-OST0000 complete [ 2394.081473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.086968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2394.095419] Lustre: Skipped 1 previous similar message [ 2394.102178] Lustre: Skipped 1 previous similar message [ 2399.568536] Lustre: server umount lustre-MDT0000 complete [ 2401.209982] LustreError: 110547:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520044 with bad export cookie 14301508829259400629 [ 2401.213563] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2401.214568] LustreError: 110547:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2407.379554] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2408.597325] Key type lgssc unregistered [ 2408.725402] LNet: 114241:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.731445] LNet: Removed LNI 192.168.203.115@tcp [ 2409.112853] Key type .llcrypt unregistered [ 2409.115359] Key type ._llcrypt unregistered [ 2420.470039] alg: No test for adler32 (adler32-zlib) [ 2421.221398] Key type ._llcrypt registered [ 2421.222913] Key type .llcrypt registered [ 2421.266637] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2421.702753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.871808] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2421.954185] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2421.956513] LNet: Accept secure, port 988 [ 2423.543119] Key type lgssc registered [ 2423.927089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2427.114944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.350259] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.377174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.179799] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2433.504389] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.780802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.940620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2435.912138] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2439.307383] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2441.835287] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2444.684664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2444.802490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.675637] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2450.000291] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2458.153263] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2462.738067] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2463.752333] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2465.761625] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2465.768291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.776686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.321054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.321516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.949671] Lustre: server umount lustre-OST0000 complete [ 2473.440616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.441418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.445372] Lustre: Skipped 1 previous similar message [ 2473.447959] Lustre: Skipped 1 previous similar message [ 2478.560182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.565241] Lustre: Skipped 2 previous similar messages [ 2478.725583] Lustre: server umount lustre-MDT0000 complete [ 2480.190720] LustreError: 116486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520123 with bad export cookie 3839955694572980869 [ 2480.191789] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.197344] LustreError: 116486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2480.317186] Lustre: server umount lustre-MDT0001 complete [ 2485.658476] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2486.808973] Key type lgssc unregistered [ 2486.936204] LNet: 119478:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2486.939318] LNet: Removed LNI 192.168.203.115@tcp [ 2487.263735] Key type .llcrypt unregistered [ 2487.265600] Key type ._llcrypt unregistered [ 2494.787588] alg: No test for adler32 (adler32-zlib) [ 2495.539326] Key type ._llcrypt registered [ 2495.540640] Key type .llcrypt registered [ 2495.584464] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2495.927524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2496.161937] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2496.261420] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2496.263658] LNet: Accept secure, port 988 [ 2497.847101] Key type lgssc registered [ 2498.232101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2501.460416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.650066] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.682499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.529248] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2507.745117] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.733139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.927452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2511.122415] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2515.268770] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2518.162504] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2521.427548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2521.568084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.696534] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2527.469796] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2532.838289] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2537.953567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.954661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.955234] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2537.958206] Lustre: Skipped 1 previous similar message [ 2537.968685] Lustre: Skipped 1 previous similar message [ 2542.461411] Lustre: server umount lustre-OST0000 complete [ 2544.608604] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2544.612170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.618896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.192104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.194547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.669705] Lustre: server umount lustre-MDT0000 complete [ 2552.485587] LustreError: 120720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520195 with bad export cookie 14416995580410023543 [ 2552.491498] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.496077] LustreError: 120720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2552.668101] Lustre: server umount lustre-MDT0001 complete [ 2558.392023] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2559.620509] Key type lgssc unregistered [ 2559.744422] LNet: 124391:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2559.748175] LNet: Removed LNI 192.168.203.115@tcp [ 2560.108638] Key type .llcrypt unregistered [ 2560.109963] Key type ._llcrypt unregistered [ 2571.157066] alg: No test for adler32 (adler32-zlib) [ 2571.911343] Key type ._llcrypt registered [ 2571.912373] Key type .llcrypt registered [ 2571.961682] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2572.329253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2572.576579] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2572.661361] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2572.663625] LNet: Accept secure, port 988 [ 2574.255139] Key type lgssc registered [ 2574.714854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2578.146056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.313030] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.333233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.036508] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2584.545071] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.675904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.861463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2586.984176] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2590.505885] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2593.014922] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2596.045447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.188178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.014627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2601.429684] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2609.510206] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2614.699262] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2616.070885] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2616.073914] Lustre: Skipped 2 previous similar messages [ 2617.768620] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2617.771298] Lustre: Skipped 2 previous similar messages [ 2622.456304] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2622.467619] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2625.869648] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2625.872492] Lustre: Skipped 2 previous similar messages [ 2629.498091] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 2635.232469] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2635.235525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.237980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.246626] Lustre: Skipped 1 previous similar message [ 2637.724900] Lustre: server umount lustre-OST0000 complete [ 2640.352848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.353922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2640.358041] Lustre: Skipped 1 previous similar message [ 2640.362332] Lustre: Skipped 2 previous similar messages [ 2645.459352] Lustre: server umount lustre-MDT0000 complete [ 2645.472359] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.481189] LustreError: Skipped 1 previous similar message [ 2647.290768] LustreError: 125832:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520290 with bad export cookie 13990769470221513171 [ 2647.291657] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.297720] LustreError: 125832:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2647.470269] Lustre: server umount lustre-MDT0001 complete [ 2653.728502] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2655.129083] Key type lgssc unregistered [ 2655.266339] LNet: 130674:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2655.270145] LNet: Removed LNI 192.168.203.115@tcp [ 2655.643720] Key type .llcrypt unregistered [ 2655.645225] Key type ._llcrypt unregistered [ 2669.804641] alg: No test for adler32 (adler32-zlib) [ 2670.556380] Key type ._llcrypt registered [ 2670.558035] Key type .llcrypt registered [ 2670.611370] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2671.164934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.314393] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2671.402708] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2671.404835] LNet: Accept secure, port 988 [ 2673.008377] Key type lgssc registered [ 2673.488575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2676.893028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.086295] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.109067] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2678.122686] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 2678.137958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.077385] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2683.360965] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.941268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.092908] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2684.109259] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 2684.136533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.004668] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2689.252552] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2691.679926] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2694.746943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.908085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.616020] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2699.883571] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2708.197890] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 2714.591900] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.594594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.598365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.133872] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2716.138579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.144186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.690336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.697759] Lustre: Skipped 1 previous similar message [ 2720.039778] Lustre: server umount lustre-OST0000 complete [ 2723.810385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.810988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2723.815944] Lustre: Skipped 1 previous similar message [ 2723.825373] Lustre: Skipped 1 previous similar message [ 2727.854537] Lustre: server umount lustre-MDT0000 complete [ 2728.933028] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.938589] LustreError: Skipped 1 previous similar message [ 2729.923277] LustreError: 132263:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520373 with bad export cookie 5465174393874261709 [ 2729.925149] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.929695] LustreError: 132263:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2730.115906] Lustre: server umount lustre-MDT0001 complete [ 2737.012094] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2738.256568] Key type lgssc unregistered [ 2738.374414] LNet: 136008:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2738.378527] LNet: Removed LNI 192.168.203.115@tcp [ 2738.757982] Key type .llcrypt unregistered [ 2738.759415] Key type ._llcrypt unregistered [ 2751.710483] alg: No test for adler32 (adler32-zlib) [ 2752.464363] Key type ._llcrypt registered [ 2752.465943] Key type .llcrypt registered [ 2752.508083] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2759.351440] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2759.807423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.987909] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2760.089300] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2760.091865] LNet: Accept secure, port 988 [ 2761.671098] Key type lgssc registered [ 2762.117327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2766.150444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.403514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.224681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2775.188208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2778.296528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2778.322781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.443483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2779.463065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2779.511585] Lustre: lustre-MDT0000: new disk, initializing [ 2779.566547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.585257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2781.589940] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2787.576658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2787.642163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.698672] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2787.728936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2787.733727] Lustre: Skipped 1 previous similar message [ 2787.783116] Lustre: lustre-MDT0001: new disk, initializing [ 2787.824125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.841053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2787.851176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2790.208677] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2795.554372] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.652899] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2802.332944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2802.406922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.416249] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2802.599864] Lustre: lustre-OST0000: new disk, initializing [ 2802.602896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2802.661467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.164349] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2810.263086] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2812.445053] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2812.931237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2812.935553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2813.569132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2815.804604] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2815.911358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2818.019340] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2818.019356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.021920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2818.030390] Lustre: Skipped 1 previous similar message [ 2823.027571] Lustre: server umount lustre-OST0000 complete [ 2828.256133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.256567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.262022] Lustre: Skipped 1 previous similar message [ 2828.265418] Lustre: Skipped 2 previous similar messages [ 2830.978184] Lustre: server umount lustre-MDT0000 complete [ 2832.477212] LustreError: 141605:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520475 with bad export cookie 16849014515735685432 [ 2832.479850] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.482134] LustreError: 141605:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2832.616148] Lustre: server umount lustre-MDT0001 complete [ 2835.940753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.130405] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.163449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.000607] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2841.313364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.457574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.085834] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2846.045628] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2848.425172] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2851.134742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.255554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.893831] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2855.938875] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2857.693753] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2857.830205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2859.886807] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2860.027405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2864.016017] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2864.045664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2864.048800] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2864.111791] Lustre: lustre-OST0001: new disk, initializing [ 2864.114199] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2864.298740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2864.301912] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2865.767877] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2869.683322] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2869.773547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2871.481927] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2871.564421] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2873.883420] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9eb8c6a23800.ost_server_uuid 40 [ 2874.520165] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9eb8c6a23800.ost_server_uuid in FULL state after 0 sec [ 2879.969684] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.970242] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.973628] LustreError: Skipped 2 previous similar messages [ 2879.974573] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2879.978562] Lustre: Skipped 1 previous similar message [ 2883.120727] Lustre: server umount lustre-OST0001 complete [ 2885.089018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.089724] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.091356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.091362] Lustre: Skipped 1 previous similar message [ 2885.093989] Lustre: Skipped 1 previous similar message [ 2885.103252] LustreError: Skipped 1 previous similar message [ 2890.208960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2890.209852] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.211124] Lustre: Skipped 2 previous similar messages [ 2890.217836] LustreError: Skipped 1 previous similar message [ 2890.779116] Lustre: server umount lustre-OST0000 complete [ 2892.767795] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2892.770287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.107265] LustreError: 143864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520543 with bad export cookie 16849014515735686286 [ 2900.108853] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.114147] LustreError: 143864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2900.263174] Lustre: server umount lustre-MDT0001 complete [ 2900.264959] Lustre: Skipped 1 previous similar message [ 2909.495678] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2910.588280] Key type lgssc unregistered [ 2910.745743] LNet: 150526:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.751062] LNet: Removed LNI 192.168.203.115@tcp [ 2911.188988] Key type .llcrypt unregistered [ 2911.190748] Key type ._llcrypt unregistered [ 2924.331846] alg: No test for adler32 (adler32-zlib) [ 2925.083321] Key type ._llcrypt registered [ 2925.084813] Key type .llcrypt registered [ 2925.127185] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2931.902054] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2932.256758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2932.490406] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 2932.576935] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2932.578874] LNet: Accept secure, port 988 [ 2934.167089] Key type lgssc registered [ 2934.548280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2938.184705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2940.804298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.451770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2946.049590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2948.746432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2948.776142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.873566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2949.894363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2949.941517] Lustre: lustre-MDT0000: new disk, initializing [ 2949.981264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.989450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2951.585566] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2956.141543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.169376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.206249] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2956.223686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2956.226398] Lustre: Skipped 1 previous similar message [ 2956.263441] Lustre: lustre-MDT0001: new disk, initializing [ 2956.293476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2956.306094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2956.309539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2957.932193] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2962.295529] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2964.493648] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2967.079870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2967.112349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.116773] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2967.236375] Lustre: lustre-OST0000: new disk, initializing [ 2967.238719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2967.273345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.970833] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2973.418869] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2975.160824] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2975.741358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2975.745403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2976.265428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2977.890738] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2977.973178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2980.834408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.836913] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.839229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.841743] LustreError: Skipped 1 previous similar message [ 2980.844267] Lustre: Skipped 1 previous similar message [ 2984.789676] Lustre: server umount lustre-OST0000 complete [ 2986.976155] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2986.979390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.984249] Lustre: Skipped 1 previous similar message [ 2986.986390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.072436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.074164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.386618] Lustre: server umount lustre-MDT0000 complete [ 2993.862970] LustreError: 155465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520636 with bad export cookie 3818436692038491750 [ 2993.865353] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2993.866826] LustreError: 155465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2993.990306] Lustre: server umount lustre-MDT0001 complete [ 2997.254150] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 06:23:59 (1774520639) [ 3003.213440] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3006.372848] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3009.888546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3012.366842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.703172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3017.095377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3019.773642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3019.793462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.885204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3019.897348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3019.937974] Lustre: lustre-MDT0000: new disk, initializing [ 3019.969942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3019.977286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3021.591208] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3026.074356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.099793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.140498] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3026.158775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3026.161939] Lustre: Skipped 1 previous similar message [ 3026.199232] Lustre: lustre-MDT0001: new disk, initializing [ 3026.231778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3026.245631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3026.249492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3027.766079] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3032.106602] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3034.367664] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3036.998740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3037.046318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3037.050111] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3037.145058] Lustre: lustre-OST0000: new disk, initializing [ 3037.147200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3037.176645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.904078] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3043.601744] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3046.396863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3046.400353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3051.173842] Lustre: Failing over lustre-OST0000 [ 3051.250288] Lustre: server umount lustre-OST0000 complete [ 3058.143145] Lustre: 152840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774520694/real 1774520694] req@000000002ca5b982 x1860719686479744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774520701 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3058.150683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.167151] Lustre: 152839:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774520694/real 1774520694] req@00000000f820b28c x1860719686479360/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1774520701 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3059.176651] Lustre: 152839:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3062.239205] Lustre: 152840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774520698/real 1774520698] req@000000005fe9eaca x1860719686480000/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1774520705 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3062.246849] Lustre: 152840:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3065.034577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.152031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.157756] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3065.823125] Lustre: 152840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774520701/real 1774520701] req@00000000948ee3a7 x1860719686481024/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774520708 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3065.831252] Lustre: 152840:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3066.775480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3068.250039] Lustre: Failing over lustre-MDT0000 [ 3068.407829] Lustre: server umount lustre-MDT0000 complete [ 3068.728138] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3068.735909] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.153153] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.160206] Lustre: Skipped 3 previous similar messages [ 3073.850415] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.861466] LustreError: Skipped 3 previous similar messages [ 3075.552485] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.557342] LustreError: Skipped 5 previous similar messages [ 3076.575240] Lustre: 152840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774520712/real 1774520712] req@00000000663754e5 x1860719686484224/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774520719 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3076.591618] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3080.609089] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.613887] LustreError: Skipped 6 previous similar messages [ 3082.146548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.657048] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x34fdcff0c245a1ae to 0x34fdcff0c245a64d [ 3082.663236] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [ 3082.745946] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3082.798839] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3084.477309] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3087.166959] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3087.843735] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3087.854542] Lustre: lustre-OST0000: Recovery over after 0:19, of 3 clients 3 recovered and 0 were evicted. [ 3087.879114] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3092.960639] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.961054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.964465] LustreError: Skipped 1 previous similar message [ 3092.970569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.974311] Lustre: Skipped 2 previous similar messages [ 3095.191725] Lustre: server umount lustre-OST0000 complete [ 3098.083635] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3098.084239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3098.089504] Lustre: Skipped 1 previous similar message [ 3103.199848] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.204282] LustreError: Skipped 9 previous similar messages [ 3104.181195] LustreError: 161596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774520747 with bad export cookie 3818436692038493773 [ 3104.184460] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.187715] LustreError: 161596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3109.177478] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3110.233769] Key type lgssc unregistered [ 3110.354484] LNet: 167444:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.358322] LNet: Removed LNI 192.168.203.115@tcp [ 3110.676360] Key type .llcrypt unregistered [ 3110.681139] Key type ._llcrypt unregistered [ 3120.186495] alg: No test for adler32 (adler32-zlib) [ 3120.938405] Key type ._llcrypt registered [ 3120.939734] Key type .llcrypt registered [ 3120.985461] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3121.321895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3121.587743] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 3121.694534] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3121.697040] LNet: Accept secure, port 988 [ 3123.279113] Key type lgssc registered [ 3123.662117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3126.644213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.842666] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.859602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.429887] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3132.464424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.605304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3134.153881] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3136.838416] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3138.896807] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3141.263451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.380813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.878315] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3143.460710] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3145.753120] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3284.338848] Lustre: lustre-MDT0000: Client 15beac40-4bf4-4438-90d4-154305076750 (at 192.168.203.15@tcp) reconnecting [ 3357.003343] Lustre: lustre-MDT0000: Client 15beac40-4bf4-4438-90d4-154305076750 (at 192.168.203.15@tcp) reconnecting [ 3375.802699] Lustre: Failing over lustre-MDT0000 [ 3375.936112] Lustre: server umount lustre-MDT0000 complete [ 3377.472312] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.146575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.151052] Lustre: Skipped 2 previous similar messages [ 3382.586416] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.602209] LustreError: Skipped 3 previous similar messages [ 3385.311207] Lustre: 168227:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774521021/real 1774521021] req@00000000ebfd9022 x1860719884964864/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774521028 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3385.320352] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3385.328235] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.335660] LustreError: Skipped 6 previous similar messages [ 3389.757169] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3391.628852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.026865] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.038928] LustreError: Skipped 11 previous similar messages [ 3402.723501] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x6f5e8bdfbfa2cce1 to 0x6f5e8bdfbfa53902 [ 3402.728446] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 3402.857524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.899174] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3405.025859] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3406.138734] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3408.359332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3408.377287] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3408.414632] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 3412.242677] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3413.262806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3428.832149] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3428.838537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.844965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3428.853758] Lustre: Skipped 2 previous similar messages [ 3428.870086] Lustre: Skipped 2 previous similar messages [ 3433.952816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.959867] Lustre: Skipped 2 previous similar messages [ 3439.073259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.082468] Lustre: Skipped 2 previous similar messages [ 3442.143204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3442.214977] Lustre: server umount lustre-MDT0000 complete [ 3444.194016] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.207403] LustreError: Skipped 8 previous similar messages [ 3444.629982] LustreError: 168893:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521087 with bad export cookie 8025005379130046722 [ 3444.640043] LustreError: 168893:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3444.640341] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.313261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.318721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3450.894486] Lustre: server umount lustre-MDT0001 complete [ 3459.660937] Lustre: server umount lustre-OST0000 complete [ 3465.149467] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3471.103427] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3476.750173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3480.657530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3484.741600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3488.362202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.927723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.984972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.113910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3492.131204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3492.178529] Lustre: lustre-MDT0000: new disk, initializing [ 3492.224136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.233599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3494.673236] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3501.579152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.616836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.671137] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3501.694355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3501.696583] Lustre: Skipped 1 previous similar message [ 3501.747160] Lustre: lustre-MDT0001: new disk, initializing [ 3501.800250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.827634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3501.839149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3504.729111] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3511.982539] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3516.023148] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3520.491571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3520.532975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.535792] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3520.694889] Lustre: lustre-OST0000: new disk, initializing [ 3520.697892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3520.766466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.509562] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3526.216037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3526.230336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3530.023381] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3532.663656] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3532.794626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3535.381402] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3535.499285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3536.865335] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.868376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.871543] LustreError: Skipped 1 previous similar message [ 3536.882822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.886547] Lustre: Skipped 2 previous similar messages [ 3542.635681] Lustre: server umount lustre-OST0000 complete [ 3547.104874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.109754] Lustre: Skipped 2 previous similar messages [ 3547.109757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.109765] Lustre: Skipped 2 previous similar messages [ 3550.793379] Lustre: server umount lustre-MDT0000 complete [ 3552.224306] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.234720] LustreError: Skipped 3 previous similar messages [ 3553.249867] LustreError: 176640:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521196 with bad export cookie 8025005379130047436 [ 3553.256848] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.268457] LustreError: 176640:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3558.671351] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:33:20 (1774521200) [ 3568.999142] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3574.565774] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3579.510726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3583.279251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3586.705279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3590.064566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.831411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.877595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.997529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3594.009588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3594.055866] Lustre: lustre-MDT0000: new disk, initializing [ 3594.114935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.127236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3596.416720] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3603.130586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3603.174913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.227430] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3603.262212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3603.266708] Lustre: Skipped 1 previous similar message [ 3603.335808] Lustre: lustre-MDT0001: new disk, initializing [ 3603.428319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3603.439161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3606.453663] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3613.653413] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3617.257205] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3622.077606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3622.127490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.137852] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3622.283229] Lustre: lustre-OST0000: new disk, initializing [ 3622.284938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3625.395781] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3632.115111] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3632.660843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3632.667368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3642.849371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.853890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.860136] Lustre: Skipped 1 previous similar message [ 3648.448906] Lustre: server umount lustre-OST0000 complete [ 3648.453872] Lustre: Skipped 1 previous similar message [ 3665.375760] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3668.304589] LustreError: 182758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521311 with bad export cookie 8025005379130048290 [ 3668.305977] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.313400] LustreError: 182758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3673.877892] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:35:16 (1774521316) [ 3684.038829] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3689.632681] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3696.264927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3699.768182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3703.004475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.167864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3711.666588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3711.710188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.852768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3711.871401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3711.922154] Lustre: lustre-MDT0000: new disk, initializing [ 3711.972125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.976581] Lustre: Skipped 2 previous similar messages [ 3711.985620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3714.129717] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3720.514777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3720.573178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.625335] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3720.744470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3722.961710] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3728.667928] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3731.857854] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3735.357423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3735.389487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.393432] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3737.769658] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3741.699617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3743.001907] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3746.787060] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3746.787309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.788454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.788459] Lustre: Skipped 9 previous similar messages [ 3746.793106] LustreError: Skipped 1 previous similar message [ 3746.797607] Lustre: Skipped 3 previous similar messages [ 3748.252687] Lustre: server umount lustre-OST0000 complete [ 3748.255596] Lustre: Skipped 2 previous similar messages [ 3751.392690] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3757.024602] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.035340] LustreError: Skipped 1 previous similar message [ 3758.203269] LustreError: 189220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521401 with bad export cookie 8025005379130049368 [ 3758.205068] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3758.209616] LustreError: 189220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3764.136655] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3765.405939] Key type lgssc unregistered [ 3765.534458] LNet: 193190:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3765.538145] LNet: Removed LNI 192.168.203.115@tcp [ 3765.905458] Key type .llcrypt unregistered [ 3765.907089] Key type ._llcrypt unregistered [ 3778.356896] alg: No test for adler32 (adler32-zlib) [ 3779.108360] Key type ._llcrypt registered [ 3779.109941] Key type .llcrypt registered [ 3779.173274] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3779.682230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3780.006123] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 3780.108472] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3780.112402] LNet: Accept secure, port 988 [ 3781.719110] Key type lgssc registered [ 3782.352860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3786.543507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3787.736395] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.754296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.870566] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3792.871744] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.840717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.111333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3797.691068] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3801.712827] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3804.589239] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3808.036190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3808.207030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.646969] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3814.334223] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3817.455347] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3832.803489] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.807168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.810394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.820108] Lustre: Skipped 1 previous similar message [ 3837.921661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.924572] Lustre: Skipped 2 previous similar messages [ 3838.511324] Lustre: server umount lustre-OST0000 complete [ 3841.505067] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3841.510436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.517427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.048892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.544284] Lustre: server umount lustre-MDT0000 complete [ 3848.161467] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.170693] LustreError: Skipped 1 previous similar message [ 3848.626908] LustreError: 197521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521491 with bad export cookie 10193676424723486895 [ 3848.627489] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.632661] LustreError: 197521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.785823] Lustre: server umount lustre-MDT0001 complete [ 3855.411073] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3856.837875] Key type lgssc unregistered [ 3857.000493] LNet: 198326:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.004538] LNet: Removed LNI 192.168.203.115@tcp [ 3857.435465] Key type .llcrypt unregistered [ 3857.437031] Key type ._llcrypt unregistered [ 3870.060985] alg: No test for adler32 (adler32-zlib) [ 3870.818142] Key type ._llcrypt registered [ 3870.821376] Key type .llcrypt registered [ 3870.877613] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3871.531803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3871.747490] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 3871.870703] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3871.877367] LNet: Accept secure, port 988 [ 3873.487124] Key type lgssc registered [ 3874.107900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3878.760587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.034864] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.064600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.824239] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3885.537740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.981692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.183281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3890.978362] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3895.530168] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3899.344254] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3903.219403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.389177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.681691] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3909.501479] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3910.571345] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 3919.840189] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3919.845605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.853102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.867720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.868558] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.873453] Lustre: Skipped 1 previous similar message [ 3920.883825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.727405] Lustre: server umount lustre-OST0000 complete [ 3928.458467] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9eb8d04c8000.ost_server_uuid 40 [ 3929.448901] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3942.372910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.374095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.379193] Lustre: Skipped 1 previous similar message [ 3942.384765] Lustre: Skipped 2 previous similar messages [ 3947.030128] Lustre: server umount lustre-MDT0000 complete [ 3947.487967] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.492648] LustreError: Skipped 1 previous similar message [ 3948.950475] LustreError: 202895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521592 with bad export cookie 11088246157681783709 [ 3948.951299] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.957750] LustreError: 202895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3949.125189] Lustre: server umount lustre-MDT0001 complete [ 3953.436077] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:39:55 (1774521595) [ 3956.570115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.779259] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.809090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.033287] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3961.823987] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.237539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.414553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.476779] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3969.677288] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3972.990462] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3977.115904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.294713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.608697] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3984.254707] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3986.537954] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3988.141694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3988.197785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.201788] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3988.278329] Lustre: lustre-OST0001: new disk, initializing [ 3988.281169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3990.735261] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3994.639574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3994.653906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3997.368358] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3999.202725] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.208105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.213390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.833766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.837493] Lustre: Skipped 3 previous similar messages [ 4005.106867] Lustre: server umount lustre-OST0000 complete [ 4008.259391] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.061852] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4009.194652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4011.763551] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4011.901692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4025.314533] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.317831] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4025.318289] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.318293] Lustre: Skipped 1 previous similar message [ 4025.319363] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4025.329093] LustreError: Skipped 9 previous similar messages [ 4025.334088] LustreError: Skipped 1 previous similar message [ 4029.614575] Lustre: server umount lustre-OST0001 complete [ 4035.041665] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4035.048738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.056815] Lustre: Skipped 1 previous similar message [ 4041.696497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.698648] Lustre: Skipped 6 previous similar messages [ 4046.303094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4046.374177] Lustre: server umount lustre-MDT0000 complete [ 4048.452711] LustreError: 203744:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521691 with bad export cookie 11088246157681784773 [ 4048.454749] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4048.459034] LustreError: 203744:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4052.314283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4054.485325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.866996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4059.246026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.900753] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:41:45 (1774521705) [ 4066.653087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.815264] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4066.830061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4066.929630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.933930] Lustre: Skipped 1 previous similar message [ 4069.202807] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4073.843033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.901771] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4073.924758] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4076.474898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4080.946852] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4083.885943] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4087.289785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4087.397760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4089.624788] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4089.630886] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 4093.146467] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4096.059202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.100797] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4098.278353] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4098.939793] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 4102.123316] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4104.165219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4104.166816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4104.172354] Lustre: Skipped 2 previous similar messages [ 4109.491683] Lustre: server umount lustre-OST0000 complete [ 4109.493930] Lustre: Skipped 1 previous similar message [ 4112.697263] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.707553] LustreError: Skipped 2 previous similar messages [ 4124.640159] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4124.644291] LustreError: Skipped 1 previous similar message [ 4138.209445] LustreError: 210177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521781 with bad export cookie 11088246157681786138 [ 4138.211721] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.220111] LustreError: 210177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4141.514532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4143.808800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4145.822862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4148.211234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4151.957156] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:43:14 (1774521794) [ 4160.544191] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4165.008188] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4170.438068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4173.833789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4176.885233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4180.211748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.676445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.719376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.826881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4183.841514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4183.902523] Lustre: lustre-MDT0000: new disk, initializing [ 4183.963051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.966635] Lustre: Skipped 3 previous similar messages [ 4183.982578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4186.000734] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4191.529705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4191.560838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.614877] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4191.643353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4191.646199] Lustre: Skipped 1 previous similar message [ 4191.646373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4191.652285] Lustre: Skipped 9 previous similar messages [ 4191.688887] Lustre: lustre-MDT0001: new disk, initializing [ 4191.741398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4191.745844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4193.912229] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4199.316959] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4202.367309] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4206.256956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4206.306831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4206.310619] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4206.466961] Lustre: lustre-OST0000: new disk, initializing [ 4206.471824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4206.910433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4206.924144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4209.364457] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4215.409578] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4218.007985] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4218.133720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4220.438035] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4220.541703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4222.437553] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4222.437941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.443882] LustreError: Skipped 1 previous similar message [ 4222.451156] Lustre: Skipped 5 previous similar messages [ 4227.641324] Lustre: server umount lustre-OST0000 complete [ 4227.644548] Lustre: Skipped 3 previous similar messages [ 4237.791773] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.806457] LustreError: Skipped 8 previous similar messages [ 4238.290768] LustreError: 218383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521881 with bad export cookie 11088246157681787874 [ 4238.292827] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4238.297657] LustreError: 218383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4243.024764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.380495] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4249.722462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.876069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4249.877802] Lustre: Skipped 3 previous similar messages [ 4251.911218] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4255.878371] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4258.958483] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4262.343843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4264.731996] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4268.628719] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4270.877806] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4274.114091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4276.108489] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4276.198887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.264454] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4279.268758] LustreError: Skipped 1 previous similar message [ 4286.085977] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4288.064549] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4313.011292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4315.184366] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4318.786123] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.536842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4326.945387] Lustre: server umount lustre-OST0000 complete [ 4326.946971] Lustre: Skipped 3 previous similar messages [ 4331.488852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.496898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4331.502885] Lustre: Skipped 5 previous similar messages [ 4331.510156] Lustre: Skipped 8 previous similar messages [ 4343.263086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4345.601321] LustreError: 222718:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774521988 with bad export cookie 11088246157681788728 [ 4345.608365] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.610196] LustreError: 222718:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4350.348692] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:46:32 (1774521992) [ 4354.169474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.928464] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4361.945712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.684648] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4369.020832] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4371.955496] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4375.501204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.148839] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4382.607882] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4385.418165] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4385.563973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4387.868929] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4387.990202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4391.962778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4392.020171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4392.026116] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4392.122550] Lustre: lustre-OST0001: new disk, initializing [ 4392.124340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4392.198417] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.205457] Lustre: Skipped 5 previous similar messages [ 4394.524721] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4399.548325] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4401.719180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4401.726193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4401.743641] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4401.859640] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4404.081693] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4404.197254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4406.757185] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4411.874114] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.880582] LustreError: Skipped 5 previous similar messages [ 4413.939947] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4414.029812] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4415.926536] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4416.036743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4440.735020] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4442.866987] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4446.179384] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4446.189542] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4446.190863] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 4446.198903] Lustre: Skipped 1 previous similar message [ 4446.561479] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4447.291147] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4470.239955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.245267] Lustre: Skipped 4 previous similar messages [ 4473.133050] Lustre: server umount lustre-OST0000 complete [ 4473.134902] Lustre: Skipped 4 previous similar messages [ 4482.524049] LustreError: 228924:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522125 with bad export cookie 11088246157681789659 [ 4482.525428] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.530022] LustreError: 228924:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4491.578223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.509192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.271428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4496.972213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.851063] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:49:02 (1774522142) [ 4502.838985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.977766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4502.991785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4504.890180] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4508.239098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.280972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4510.096089] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4513.128762] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4515.404984] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.969728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.047731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4519.714769] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4522.648620] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4526.853226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.884119] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4528.475135] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4531.390135] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4532.046048] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4533.801491] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4533.902871] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4535.562842] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4535.660217] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4538.071743] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9eb8c7883800.ost_server_uuid 40 [ 4538.781484] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9eb8c7883800.ost_server_uuid in FULL state after 0 sec [ 4539.203644] Lustre: Permanently deactivating lustre-OST0001 [ 4539.209758] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 4539.212263] Lustre: Skipped 1 previous similar message [ 4540.304583] Lustre: Permanently reactivating lustre-OST0001 [ 4544.995446] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4544.999536] LustreError: Skipped 5 previous similar messages [ 4558.303110] Lustre: 242939:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774522195/real 1774522195] req@000000001cbefd60 x1860720671231104/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774522201 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4559.839161] Lustre: 199100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774522195/real 1774522195] req@00000000be09e5eb x1860720671230464/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1774522202 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4559.854794] Lustre: 199100:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4562.029829] LustreError: 237884:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522205 with bad export cookie 11088246157681790933 [ 4562.032881] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.035788] LustreError: 237884:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4564.493086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4566.310994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4567.996919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4569.478911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4572.193678] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:50:14 (1774522214) [ 4572.568827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4575.128634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.249085] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4575.264544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4575.266903] Lustre: Skipped 5 previous similar messages [ 4577.030503] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4580.335534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.391478] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4582.251594] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4585.297511] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4587.765208] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4590.277738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4590.310942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4590.389472] Lustre: Permanently deactivating lustre-OST0000 [ 4592.119018] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4592.186382] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4595.225054] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4598.071365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.979913] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4603.159712] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4603.412979] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 4609.379875] Lustre: Permanently reactivating lustre-OST0000 [ 4609.385814] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 4609.388374] Lustre: Skipped 4 previous similar messages [ 4612.920414] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4612.923984] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.932988] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4612.949621] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4612.963029] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 4618.723076] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4618.725482] Lustre: Skipped 26 previous similar messages [ 4635.463924] LustreError: 248303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522278 with bad export cookie 11088246157681792522 [ 4635.466516] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.472609] LustreError: 248303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4641.067658] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4642.138886] Key type lgssc unregistered [ 4642.250437] LNet: 249616:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4642.253693] LNet: Removed LNI 192.168.203.115@tcp [ 4642.594585] Key type .llcrypt unregistered [ 4642.595916] Key type ._llcrypt unregistered [ 4651.961790] alg: No test for adler32 (adler32-zlib) [ 4652.713431] Key type ._llcrypt registered [ 4652.714815] Key type .llcrypt registered [ 4652.776224] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4653.252545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4653.404151] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 4653.486912] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4653.489362] LNet: Accept secure, port 988 [ 4655.080270] Key type lgssc registered [ 4655.471965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4656.733299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4659.211891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.373080] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.393781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.046869] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4665.259939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4665.297480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.343429] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4665.358817] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4665.362358] Lustre: Permanently deactivating lustre-MDT0001 [ 4665.496247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4667.259046] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4669.101903] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4669.104688] LustreError: 251158:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4669.110046] LustreError: 251158:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4669.115206] LustreError: 251158:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 4670.116600] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4672.521351] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4675.236158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.368762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.085137] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4680.503386] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4681.514310] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.519913] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 4683.367943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4683.452243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4685.101757] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4686.130156] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 4688.379530] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4690.215785] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 4690.215913] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 4694.380827] Lustre: Permanently reactivating lustre-MDT0001 [ 4694.388623] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 4694.391181] Lustre: Skipped 3 previous similar messages [ 4697.730608] Lustre: lustre-MDT0001: Client e23633f1-13b2-4926-b152-b6088d2210b4 (at 192.168.203.15@tcp) reconnecting [ 4701.095795] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4702.080490] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4705.946056] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 4706.081084] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4710.423122] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 4710.724518] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4714.073732] Lustre: Permanently deactivating lustre-MDT0001 [ 4714.098861] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 4714.108456] Lustre: Skipped 2 previous similar messages [ 4720.231480] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4736.487419] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.489251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.493514] Lustre: Skipped 2 previous similar messages [ 4736.502667] Lustre: Skipped 3 previous similar messages [ 4739.396343] Lustre: server umount lustre-MDT0000 complete [ 4741.603329] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.612918] LustreError: Skipped 3 previous similar messages [ 4744.217351] LustreError: 251109:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522387 with bad export cookie 13291326001047477825 [ 4744.226309] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4744.230873] LustreError: 251109:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4744.552534] Lustre: server umount lustre-MDT0001 complete [ 4750.013773] Lustre: server umount lustre-OST0000 complete [ 4766.136079] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:53:27 (1774522407) [ 4775.121967] Lustre: server umount lustre-OST0001 complete [ 4781.096690] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4788.461963] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4795.363077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4801.022810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4805.691490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4809.739390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4814.434971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4814.489732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.661455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4814.667426] Lustre: Skipped 2 previous similar messages [ 4814.692871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4814.775908] Lustre: lustre-MDT0000: new disk, initializing [ 4814.873657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.890614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4818.126047] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4826.522394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4826.589489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.632861] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4826.662374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4826.665937] Lustre: Skipped 1 previous similar message [ 4826.723263] Lustre: lustre-MDT0001: new disk, initializing [ 4826.836524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4826.885534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4826.901591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4830.604719] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4838.556965] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4843.080472] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4847.923609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4847.998299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.007510] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4848.183938] Lustre: lustre-OST0000: new disk, initializing [ 4848.187845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4848.251459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.246420] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4857.921989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4857.938660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4859.194651] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4861.020541] LustreError: 259634:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4868.625783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4868.688053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4868.693377] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4868.800524] Lustre: lustre-OST0001: new disk, initializing [ 4868.805493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4871.114680] LustreError: 259634:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 4871.128994] LustreError: 259633:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4872.343766] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4878.907994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4878.915873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4879.619874] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4880.852757] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4881.183163] LustreError: 259633:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 4883.945602] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.957402] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4883.963148] Lustre: Skipped 2 previous similar messages [ 4883.975706] Lustre: Skipped 1 previous similar message [ 4885.848841] Lustre: lustre-OST0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 4888.949419] Lustre: server umount lustre-OST0001 complete [ 4889.067448] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.077501] LustreError: Skipped 1 previous similar message [ 4894.181068] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.194067] LustreError: Skipped 2 previous similar messages [ 4904.417161] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4904.419073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.428100] LustreError: Skipped 1 previous similar message [ 4904.437818] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.440065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4904.457198] LustreError: Skipped 3 previous similar messages [ 4909.560270] Lustre: server umount lustre-OST0000 complete [ 4914.149594] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4914.160920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.171936] Lustre: Skipped 1 previous similar message [ 4914.176927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4914.190767] Lustre: Skipped 1 previous similar message [ 4920.802059] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.815905] LustreError: Skipped 3 previous similar messages [ 4922.010426] LustreError: 259617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522565 with bad export cookie 13291326001047480590 [ 4922.011440] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4922.017308] LustreError: 259617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4928.435723] Lustre: server umount lustre-MDT0001 complete [ 4928.437441] Lustre: Skipped 1 previous similar message [ 4938.728335] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4940.622608] Key type lgssc unregistered [ 4940.835468] LNet: 264821:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4940.847954] LNet: Removed LNI 192.168.203.115@tcp [ 4941.376772] Key type .llcrypt unregistered [ 4941.378776] Key type ._llcrypt unregistered [ 4967.373204] alg: No test for adler32 (adler32-zlib) [ 4968.127383] Key type ._llcrypt registered [ 4968.133144] Key type .llcrypt registered [ 4968.237189] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4981.388393] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4982.521193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4982.991965] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 4983.141681] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4983.144409] LNet: Accept secure, port 988 [ 4984.775271] Key type lgssc registered [ 4985.731612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4993.099756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4998.906270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5003.889306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5009.846182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5015.779246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5015.858620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.147260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5017.180323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5017.257479] Lustre: lustre-MDT0000: new disk, initializing [ 5017.346407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.371172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5021.319518] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5031.667389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5031.758263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.859086] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5031.913880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5031.919557] Lustre: Skipped 1 previous similar message [ 5032.013240] Lustre: lustre-MDT0001: new disk, initializing [ 5032.106531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.146419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5032.152109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5036.166373] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5044.720655] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5050.504554] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5056.243752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5056.297766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.302784] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5056.486490] Lustre: lustre-OST0000: new disk, initializing [ 5056.497926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5056.563315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.235781] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5065.281670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5065.296586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5068.190508] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5071.710956] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5071.858698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5075.265237] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5075.428309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5081.056629] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5081.064522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.074770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5081.084854] Lustre: Skipped 1 previous similar message [ 5083.327283] Lustre: server umount lustre-OST0000 complete [ 5088.736584] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.741985] LustreError: Skipped 1 previous similar message [ 5088.746037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.766956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5088.775786] Lustre: Skipped 1 previous similar message [ 5091.296304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.297437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.327853] Lustre: Skipped 1 previous similar message [ 5092.879814] Lustre: server umount lustre-MDT0000 complete [ 5096.223342] LustreError: 268926:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522739 with bad export cookie 3883633459863707406 [ 5096.229490] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.443494] Lustre: server umount lustre-MDT0001 complete [ 5111.456527] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 06:59:11 (1774522751) [ 5121.689224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.287925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.386458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.655838] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.608485] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5132.768281] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.898184] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5141.455749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.018931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5148.491879] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5159.656458] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5167.607682] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5175.138289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.759424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.006093] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5189.764966] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5218.785154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.810364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.825179] Lustre: Skipped 1 previous similar message [ 5223.915936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5223.922247] Lustre: Skipped 1 previous similar message [ 5224.443521] Lustre: server umount lustre-OST0000 complete [ 5229.181754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5242.747121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.091941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.168197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.179880] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.180090] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 5245.195448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 5245.210127] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 5247.841672] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5255.888100] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5271.033173] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5271.034694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.040045] LustreError: Skipped 1 previous similar message [ 5271.048355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5271.070246] Lustre: Skipped 1 previous similar message [ 5271.100338] Lustre: Skipped 2 previous similar messages [ 5274.305065] Lustre: server umount lustre-OST0000 complete [ 5280.226210] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5280.233330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.732271] Lustre: server umount lustre-MDT0000 complete [ 5286.393273] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.415798] LustreError: Skipped 1 previous similar message [ 5289.059216] LustreError: 273467:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774522932 with bad export cookie 3883633459863708260 [ 5289.068055] LustreError: 273467:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5289.068374] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.023449] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5305.842838] Key type lgssc unregistered [ 5306.111316] LNet: 278954:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5306.120549] LNet: Removed LNI 192.168.203.115@tcp [ 5307.027638] Key type .llcrypt unregistered [ 5307.031129] Key type ._llcrypt unregistered [ 5333.800526] alg: No test for adler32 (adler32-zlib) [ 5334.560900] Key type ._llcrypt registered [ 5334.566694] Key type .llcrypt registered [ 5334.739404] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5335.864185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5336.844867] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 5337.131804] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5337.137986] LNet: Accept secure, port 988 [ 5338.823130] Key type lgssc registered [ 5340.108724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5347.037687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.445041] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.488827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.733469] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5353.954278] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.181687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.374468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5361.013496] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5365.355331] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5368.805186] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5372.817759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.001114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.384868] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5379.178745] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 5380.037366] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5394.913226] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.914270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.922305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5394.925263] Lustre: Skipped 1 previous similar message [ 5400.034797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5400.043100] Lustre: Skipped 1 previous similar message [ 5401.067851] Lustre: server umount lustre-OST0000 complete [ 5404.640418] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5404.642457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.647844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.650084] Lustre: Skipped 1 previous similar message [ 5409.589493] Lustre: server umount lustre-MDT0000 complete [ 5410.271737] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5410.282779] LustreError: Skipped 1 previous similar message [ 5411.637498] LustreError: 280424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523054 with bad export cookie 16850101221133899759 [ 5411.643936] LustreError: 280424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5411.644352] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.819543] Lustre: server umount lustre-MDT0001 complete [ 5419.493869] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5421.596536] Key type lgssc unregistered [ 5421.803574] LNet: 284866:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5421.810370] LNet: Removed LNI 192.168.203.115@tcp [ 5422.320482] Key type .llcrypt unregistered [ 5422.323238] Key type ._llcrypt unregistered [ 5433.111740] alg: No test for adler32 (adler32-zlib) [ 5433.864336] Key type ._llcrypt registered [ 5433.865655] Key type .llcrypt registered [ 5433.924820] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5434.512947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5434.742025] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 5434.888675] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5434.893643] LNet: Accept secure, port 988 [ 5436.520595] Key type lgssc registered [ 5437.221989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5442.290958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.583069] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.614963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5446.003872] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5448.681403] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.133025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.392750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.370430] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5459.207345] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.125888] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5467.168564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5467.388315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.663164] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5473.518902] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 5474.227799] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5492.193661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.194423] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5492.205594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.212822] Lustre: Skipped 1 previous similar message [ 5496.266915] Lustre: server umount lustre-OST0000 complete [ 5502.432555] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5502.432839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.443445] Lustre: Skipped 1 previous similar message [ 5502.454981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5502.466154] Lustre: Skipped 1 previous similar message [ 5504.801074] Lustre: server umount lustre-MDT0000 complete [ 5507.177615] LustreError: 286114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523150 with bad export cookie 12543085353996381619 [ 5507.188783] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.435226] Lustre: server umount lustre-MDT0001 complete [ 5514.958131] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5516.531979] Key type lgssc unregistered [ 5516.686450] LNet: 289970:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5516.690685] LNet: Removed LNI 192.168.203.115@tcp [ 5517.178892] Key type .llcrypt unregistered [ 5517.180842] Key type ._llcrypt unregistered [ 5527.608935] alg: No test for adler32 (adler32-zlib) [ 5528.360419] Key type ._llcrypt registered [ 5528.361983] Key type .llcrypt registered [ 5528.425655] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5529.212659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5529.475489] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 5529.685958] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5529.692303] LNet: Accept secure, port 988 [ 5531.353034] Key type lgssc registered [ 5531.981554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5536.908645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.190590] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5538.225879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.102645] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5543.391805] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.346423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.542091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5548.686712] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5552.677471] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5556.319742] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5560.554174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.751541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.331339] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5567.917783] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 5567.937319] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5578.210088] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.211625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.214543] LustreError: Skipped 1 previous similar message [ 5578.217380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5578.235748] Lustre: Skipped 1 previous similar message [ 5579.454741] Lustre: server umount lustre-OST0000 complete [ 5582.306049] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5582.310958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.321151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.328048] Lustre: Skipped 1 previous similar message [ 5583.328722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.330055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5583.348889] Lustre: Skipped 1 previous similar message [ 5588.379573] Lustre: server umount lustre-MDT0000 complete [ 5588.451286] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.460983] LustreError: Skipped 1 previous similar message [ 5590.654801] LustreError: 291216:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523233 with bad export cookie 12315731365074110132 [ 5590.658588] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5590.661156] LustreError: 291216:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5590.835411] Lustre: server umount lustre-MDT0001 complete [ 5598.802491] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5600.902941] Key type lgssc unregistered [ 5601.117601] LNet: 294889:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5601.122699] LNet: Removed LNI 192.168.203.115@tcp [ 5601.683852] Key type .llcrypt unregistered [ 5601.686609] Key type ._llcrypt unregistered [ 5615.824383] alg: No test for adler32 (adler32-zlib) [ 5616.576324] Key type ._llcrypt registered [ 5616.577270] Key type .llcrypt registered [ 5616.649058] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5617.214565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5617.400126] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 5617.494857] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5617.498530] LNet: Accept secure, port 988 [ 5619.128197] Key type lgssc registered [ 5619.676553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5624.041194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.282787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.312316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.736619] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5630.433021] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5632.569407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.854097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5635.797857] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5640.972306] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5644.874873] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5649.587838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5649.771201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.999811] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5654.913204] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 5658.645315] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5678.560906] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5678.563336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.577045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.584162] Lustre: Skipped 1 previous similar message [ 5678.593902] Lustre: Skipped 1 previous similar message [ 5679.302749] Lustre: server umount lustre-OST0000 complete [ 5683.680874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.681376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.693941] Lustre: Skipped 1 previous similar message [ 5683.700625] Lustre: Skipped 1 previous similar message [ 5687.992682] Lustre: server umount lustre-MDT0000 complete [ 5688.804640] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.814174] LustreError: Skipped 1 previous similar message [ 5690.081795] LustreError: 300025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523333 with bad export cookie 1619051607128477237 [ 5690.084135] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5690.091039] LustreError: 300025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5690.310452] Lustre: server umount lustre-MDT0001 complete [ 5697.509589] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5698.990808] Key type lgssc unregistered [ 5699.209321] LNet: 300829:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5699.215525] LNet: Removed LNI 192.168.203.115@tcp [ 5699.678585] Key type .llcrypt unregistered [ 5699.679957] Key type ._llcrypt unregistered [ 5708.786398] alg: No test for adler32 (adler32-zlib) [ 5709.537424] Key type ._llcrypt registered [ 5709.538560] Key type .llcrypt registered [ 5709.596950] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5710.428621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5710.626914] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 5710.758157] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5710.766199] LNet: Accept secure, port 988 [ 5712.391096] Key type lgssc registered [ 5713.192689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5720.110775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5721.702140] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.751216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.200069] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.175358] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5732.369886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.441133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.148415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.322725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.115360] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5762.119734] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5769.913064] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5781.281778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5781.884942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.372662] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5796.064581] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5798.260595] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 5812.708281] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5812.722059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.746442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.730779] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5813.738231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.740568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.749544] Lustre: Skipped 1 previous similar message [ 5817.827851] Lustre: server umount lustre-OST0000 complete [ 5823.972042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.978035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.981517] Lustre: Skipped 1 previous similar message [ 5823.990142] Lustre: Skipped 1 previous similar message [ 5828.433189] Lustre: server umount lustre-MDT0000 complete [ 5829.098529] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.118685] LustreError: Skipped 2 previous similar messages [ 5832.281866] LustreError: 302075:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523475 with bad export cookie 15347621155473550885 [ 5832.289658] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.540919] Lustre: server umount lustre-MDT0001 complete [ 5844.088272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5847.403063] Key type lgssc unregistered [ 5847.706632] LNet: 305936:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5847.715576] LNet: Removed LNI 192.168.203.115@tcp [ 5848.565992] Key type .llcrypt unregistered [ 5848.571499] Key type ._llcrypt unregistered [ 5866.626295] alg: No test for adler32 (adler32-zlib) [ 5867.377470] Key type ._llcrypt registered [ 5867.379033] Key type .llcrypt registered [ 5867.556465] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5868.871074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5870.020850] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 5870.346790] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5870.355154] LNet: Accept secure, port 988 [ 5872.136059] Key type lgssc registered [ 5873.424821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5881.918247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.404355] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.480139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5888.370590] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5888.483597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.604575] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.595595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.127288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5902.978712] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5910.384689] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5915.983479] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5922.119676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.412560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5926.934696] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5934.291609] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5934.707226] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 5944.289786] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.294864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.310603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.311904] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5945.313848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.322730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.351521] Lustre: Skipped 1 previous similar message [ 5950.433390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5950.445828] Lustre: Skipped 1 previous similar message [ 5955.555265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5955.562355] Lustre: Skipped 1 previous similar message [ 5956.063126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5956.205366] Lustre: server umount lustre-OST0000 complete [ 5959.655251] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5959.662415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.674382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5965.550895] Lustre: server umount lustre-MDT0000 complete [ 5965.793796] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.807527] LustreError: Skipped 1 previous similar message [ 5969.512242] LustreError: 307204:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523612 with bad export cookie 4421012999291696548 [ 5969.518190] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5969.520832] LustreError: 307204:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5969.797913] Lustre: server umount lustre-MDT0001 complete [ 5979.906191] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5982.298420] Key type lgssc unregistered [ 5982.571357] LNet: 310881:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.592396] LNet: Removed LNI 192.168.203.115@tcp [ 5983.530492] Key type .llcrypt unregistered [ 5983.532668] Key type ._llcrypt unregistered [ 5995.681445] alg: No test for adler32 (adler32-zlib) [ 5996.437750] Key type ._llcrypt registered [ 5996.440465] Key type .llcrypt registered [ 5996.586453] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6013.474378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6023.762549] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6024.946607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6025.315734] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 6025.483906] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6025.486932] LNet: Accept secure, port 988 [ 6027.119119] Key type lgssc registered [ 6028.012056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6035.705271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6041.081566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6046.279622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6051.423681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6057.475855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6057.554175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.768591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6058.789600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6058.857705] Lustre: lustre-MDT0000: new disk, initializing [ 6058.916095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.931388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6062.829855] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6071.770491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6071.832361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.943764] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6071.991806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6071.999867] Lustre: Skipped 1 previous similar message [ 6072.088535] Lustre: lustre-MDT0001: new disk, initializing [ 6072.190707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6072.215855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6072.230054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6076.331061] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6085.856411] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6091.346728] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6098.141926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6098.199739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.204311] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6098.371406] Lustre: lustre-OST0000: new disk, initializing [ 6098.375332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6098.463464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.173726] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6105.137055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6105.146132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6112.603683] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6116.734221] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6116.877753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6120.494139] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6120.629025] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6126.064879] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6126.067791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.069170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6126.084210] Lustre: Skipped 1 previous similar message [ 6128.387834] Lustre: server umount lustre-OST0000 complete [ 6133.730049] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6133.734022] LustreError: Skipped 1 previous similar message [ 6133.736361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.748412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.752324] Lustre: Skipped 1 previous similar message [ 6136.288390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6136.290805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.838969] Lustre: server umount lustre-MDT0000 complete [ 6141.256829] LustreError: 314627:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523784 with bad export cookie 2545931387612705765 [ 6141.261549] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.270966] LustreError: 314627:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6141.499894] Lustre: server umount lustre-MDT0001 complete [ 6149.107460] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:16:30 (1774523790) [ 6154.724042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.177951] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.230931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.228372] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6160.356043] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6165.478462] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.264595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.584942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6171.362320] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6177.787231] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6182.353065] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6187.942926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6188.149812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.264152] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6198.663040] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6218.208086] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.215861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.226388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.230509] Lustre: Skipped 1 previous similar message [ 6221.295457] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6222.812132] Lustre: server umount lustre-OST0000 complete [ 6226.402910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.408352] Lustre: Skipped 2 previous similar messages [ 6226.408559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6226.416057] Lustre: Skipped 3 previous similar messages [ 6231.857404] Lustre: server umount lustre-MDT0000 complete [ 6234.639907] LustreError: 322057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774523877 with bad export cookie 2545931387612706619 [ 6234.650377] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6234.659327] LustreError: 322057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6243.601500] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6245.901502] Key type lgssc unregistered [ 6246.165156] LNet: 322862:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6246.178242] LNet: Removed LNI 192.168.203.115@tcp [ 6246.812393] Key type .llcrypt unregistered [ 6246.814361] Key type ._llcrypt unregistered [ 6265.218983] alg: No test for adler32 (adler32-zlib) [ 6265.973170] Key type ._llcrypt registered [ 6265.975709] Key type .llcrypt registered [ 6266.070532] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6275.625483] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6276.366931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6276.897920] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 6277.069593] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6277.072643] LNet: Accept secure, port 988 [ 6278.703132] Key type lgssc registered [ 6279.496487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.331884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6291.176153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6295.275493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6299.832548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6303.267435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6307.093132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.801394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.875585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.038467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6313.068696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6313.132878] Lustre: lustre-MDT0000: new disk, initializing [ 6313.211231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6313.222815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6316.222866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6323.160833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6323.190577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.249576] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6323.274588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6323.277565] Lustre: Skipped 1 previous similar message [ 6323.324774] Lustre: lustre-MDT0001: new disk, initializing [ 6323.385283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6323.422386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6323.431495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6326.093562] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6331.880838] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6335.773915] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6340.079738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6340.124683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.129183] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6340.334793] Lustre: lustre-OST2710: new disk, initializing [ 6340.337973] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 6340.386990] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6343.093167] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6349.526267] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6350.371809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6350.376432] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6353.749978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6353.806342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6353.811471] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6353.879365] Lustre: lustre-OST03e8: new disk, initializing [ 6353.883472] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 6353.940189] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6356.643274] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6358.587179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6358.594934] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6361.452492] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6365.998214] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 6366.123943] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6368.915141] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 6369.080704] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6371.653698] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 6371.768134] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6374.872562] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 6375.020659] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6379.489821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.493736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6379.503015] Lustre: Skipped 3 previous similar messages [ 6379.513684] Lustre: Skipped 3 previous similar messages [ 6383.592685] Lustre: server umount lustre-MDT0000 complete [ 6384.610962] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.622682] LustreError: Skipped 3 previous similar messages [ 6385.808326] LustreError: 326800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524028 with bad export cookie 4937350030343194732 [ 6385.810807] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6385.819907] LustreError: 326800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6386.038328] Lustre: server umount lustre-MDT0001 complete [ 6394.847150] Lustre: 332290:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774524031/real 1774524031] req@0000000061edde4e x1860723193963712/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774524037 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6394.893867] Lustre: server umount lustre-OST2710 complete [ 6396.767263] Lustre: 324605:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774524032/real 1774524032] req@00000000e4ddfa8a x1860723193963840/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774524039 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6396.779156] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.264850] Lustre: server umount lustre-OST03e8 complete [ 6408.285833] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6413.364576] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6419.311445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6423.202765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6427.269073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.563043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.623401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.677046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.820934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6435.846774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6435.921347] Lustre: lustre-MDT0000: new disk, initializing [ 6435.981131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.995875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6438.722267] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6445.712924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6445.754226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.809979] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6445.833571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6445.836382] Lustre: Skipped 1 previous similar message [ 6445.887775] Lustre: lustre-MDT0001: new disk, initializing [ 6445.936464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6445.958462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6445.973522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6448.553071] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6454.618813] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6458.154980] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6462.090803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6462.135489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.140500] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6462.295338] Lustre: lustre-OST0000: new disk, initializing [ 6462.298943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6462.358191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.044356] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6468.625560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6468.637957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6471.046862] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6473.293977] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6473.392471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6475.605377] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6475.732465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6478.816792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.821091] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6478.828166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.832650] Lustre: Skipped 1 previous similar message [ 6483.941066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6483.944014] Lustre: Skipped 2 previous similar messages [ 6489.062861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6489.072658] Lustre: Skipped 1 previous similar message [ 6491.103116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6491.243810] Lustre: server umount lustre-OST0000 complete [ 6494.179760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6494.185563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6494.196135] Lustre: Skipped 1 previous similar message [ 6494.205101] Lustre: Skipped 1 previous similar message [ 6499.551204] Lustre: server umount lustre-MDT0000 complete [ 6502.010682] LustreError: 335396:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524145 with bad export cookie 4937350030343196342 [ 6502.013969] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.018381] LustreError: 335396:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6507.408453] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:22:29 (1774524149) [ 6508.247531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6509.166924] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:22:31 (1774524151) [ 6510.989229] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6514.909161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6516.680857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6518.611276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6520.523650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6521.139306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6524.670807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.844308] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6524.855215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6524.931674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6527.333956] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6532.050141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.097615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6534.636461] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6539.035803] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6542.513450] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6546.243028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6546.289490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.390934] Lustre: Denying initial registration attempt from nid 192.168.203.115@tcp, specified as failover [ 6546.393839] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6546.396601] LustreError: 343414:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 6546.402464] LustreError: 343414:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 6546.407540] LustreError: 343414:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 6546.466449] Lustre: server umount lustre-OST0000 complete [ 6546.467953] Lustre: Skipped 1 previous similar message [ 6546.469762] LustreError: 343414:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6553.059501] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.062203] LustreError: Skipped 1 previous similar message [ 6553.068422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.089277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.091647] Lustre: Skipped 2 previous similar messages [ 6564.319283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6569.374464] LustreError: 341456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524212 with bad export cookie 4937350030343197203 [ 6569.380791] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.389212] LustreError: 341456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6585.118898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6588.748947] Key type lgssc unregistered [ 6589.149176] LNet: 344490:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6589.157450] LNet: Removed LNI 192.168.203.115@tcp [ 6590.467910] Key type .llcrypt unregistered [ 6590.477425] Key type ._llcrypt unregistered [ 6608.545249] alg: No test for adler32 (adler32-zlib) [ 6609.301418] Key type ._llcrypt registered [ 6609.305715] Key type .llcrypt registered [ 6609.442394] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6610.623016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6611.442502] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 6611.652780] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6611.655874] LNet: Accept secure, port 988 [ 6613.359185] Key type lgssc registered [ 6614.406466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6620.091798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6622.925872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6626.008180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6628.738362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6629.573512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6640.251308] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6646.739265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.979972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6648.005474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6648.188376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.892562] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6658.976532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.061833] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6659.096910] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6659.290225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6662.859948] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6669.018831] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6673.332927] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6679.636799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6679.698106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.843638] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6680.036735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6683.959869] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6690.188341] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6695.906134] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.907866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.914335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6695.925864] LustreError: Skipped 1 previous similar message [ 6695.974148] Lustre: Skipped 1 previous similar message [ 6698.421479] Lustre: server umount lustre-OST0000 complete [ 6705.632239] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6705.649074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.676359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6705.678788] Lustre: Skipped 1 previous similar message [ 6711.264295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6711.267877] Lustre: Skipped 3 previous similar messages [ 6716.383121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6716.517508] Lustre: server umount lustre-MDT0000 complete [ 6717.413596] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.426985] LustreError: Skipped 1 previous similar message [ 6719.688950] LustreError: 347363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524362 with bad export cookie 4365898092666825814 [ 6719.691335] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6719.702305] LustreError: 347363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6720.018267] Lustre: server umount lustre-MDT0001 complete [ 6730.243930] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6732.859417] Key type lgssc unregistered [ 6733.115549] LNet: 351063:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6733.123281] LNet: Removed LNI 192.168.203.115@tcp [ 6734.048182] Key type .llcrypt unregistered [ 6734.049501] Key type ._llcrypt unregistered [ 6751.802338] alg: No test for adler32 (adler32-zlib) [ 6752.564811] Key type ._llcrypt registered [ 6752.568278] Key type .llcrypt registered [ 6752.710394] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6753.379113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6753.827957] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 6753.978659] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6753.991716] LNet: Accept secure, port 988 [ 6755.679803] Key type lgssc registered [ 6756.744533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6763.187500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.513822] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.566121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.634461] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.893296] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6774.754862] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.873153] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.993899] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.409240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.054928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6798.261359] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6810.280521] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6819.173669] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6827.438754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.853383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6832.862558] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6841.184623] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6850.532118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.539575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6850.548948] Lustre: Skipped 2 previous similar messages [ 6850.559312] Lustre: Skipped 2 previous similar messages [ 6853.471494] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 6855.658677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.662497] Lustre: Skipped 2 previous similar messages [ 6856.415711] Lustre: server umount lustre-MDT0000 complete [ 6858.552678] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.837801] LustreError: 352514:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524503 with bad export cookie 9167939197142130467 [ 6860.848095] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6863.729336] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 6865.890118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.264653] Lustre: server umount lustre-MDT0001 complete [ 6875.758094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6886.964212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.571258] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x7f3b102040095f23 to 0x7f3b102040097b15 [ 6894.576708] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 6894.807675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.825494] LustreError: Skipped 6 previous similar messages [ 6894.868198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.908336] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 6897.010171] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 6900.461200] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6909.334759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.881078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6914.877627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6915.057388] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6922.369181] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6924.713461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6931.451767] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6933.208133] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6940.641980] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6940.646857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.658913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6940.658925] Lustre: Skipped 2 previous similar messages [ 6940.684425] Lustre: Skipped 1 previous similar message [ 6945.506218] Lustre: server umount lustre-OST0000 complete [ 6950.884118] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6950.890881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.895747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.895756] Lustre: Skipped 1 previous similar message [ 6950.926094] Lustre: Skipped 1 previous similar message [ 6964.192093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6964.365317] Lustre: server umount lustre-MDT0000 complete [ 6967.266364] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.287673] LustreError: Skipped 9 previous similar messages [ 6969.448563] LustreError: 352512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524612 with bad export cookie 9167939197142137621 [ 6969.458517] LustreError: 352512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6969.463630] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.733757] Lustre: server umount lustre-MDT0001 complete [ 6988.266800] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6991.589725] Key type lgssc unregistered [ 6991.919438] LNet: 359064:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6991.931234] LNet: Removed LNI 192.168.203.115@tcp [ 6992.874793] Key type .llcrypt unregistered [ 6992.877444] Key type ._llcrypt unregistered [ 7017.135762] alg: No test for adler32 (adler32-zlib) [ 7017.911290] Key type ._llcrypt registered [ 7017.915253] Key type .llcrypt registered [ 7018.072936] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7019.441705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7020.065418] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 7020.287605] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7020.296642] LNet: Accept secure, port 988 [ 7022.039384] Key type lgssc registered [ 7023.179858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7031.101290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.569092] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.652127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.920727] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.097818] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7043.069870] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.164175] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.985605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.266094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7053.983644] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7062.151430] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7069.346103] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7074.783921] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7074.786872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.789211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.819843] Lustre: Skipped 1 previous similar message [ 7078.946141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7078.956017] Lustre: Skipped 2 previous similar messages [ 7084.003601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.076275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7087.371140] Lustre: server umount lustre-MDT0000 complete [ 7088.033341] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.054096] LustreError: Skipped 1 previous similar message [ 7091.592198] LustreError: 360508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524734 with bad export cookie 16168814822611220396 [ 7091.602695] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7091.604624] LustreError: 360508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7091.919397] Lustre: server umount lustre-MDT0001 complete [ 7108.304078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.702536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7108.750624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7109.035413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.677383] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7125.386796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.512190] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7125.589537] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7125.896924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7130.613165] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7139.458897] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7147.268657] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7157.605303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.790853] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7157.800229] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7157.812436] LustreError: 365221:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 7157.819181] LustreError: 365221:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 7157.823048] LustreError: 365221:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 7158.015279] Lustre: server umount lustre-OST0000 complete [ 7158.017750] LustreError: 365221:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7165.672880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.858660] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7166.012747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.056207] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7171.716152] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 7181.232994] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7189.911101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7190.012963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7190.022243] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7190.153688] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7190.251873] Lustre: lustre-OST0001: new disk, initializing [ 7190.257588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7190.386121] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7195.930519] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7198.893192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7198.904819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7205.260792] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7209.440384] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7209.446900] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.448686] LustreError: Skipped 1 previous similar message [ 7209.456972] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7209.486102] Lustre: Skipped 1 previous similar message [ 7209.508610] Lustre: Skipped 2 previous similar messages [ 7214.574078] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7214.585977] Lustre: Skipped 1 previous similar message [ 7223.266686] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7223.362058] Lustre: server umount lustre-OST0001 complete [ 7224.804802] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.834896] LustreError: Skipped 1 previous similar message [ 7229.923232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.950279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7229.953058] Lustre: Skipped 2 previous similar messages [ 7243.231658] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7243.449316] Lustre: server umount lustre-OST0000 complete [ 7248.877656] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7248.890583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.911250] Lustre: Skipped 1 previous similar message [ 7248.922561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7248.926667] Lustre: Skipped 4 previous similar messages [ 7254.218044] Lustre: server umount lustre-MDT0000 complete [ 7255.531624] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.570673] LustreError: Skipped 7 previous similar messages [ 7259.017542] LustreError: 363218:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774524902 with bad export cookie 16168814822611220949 [ 7259.025682] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.036132] LustreError: 363218:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7272.129407] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7275.577686] Key type lgssc unregistered [ 7275.959331] LNet: 368598:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7275.972885] LNet: Removed LNI 192.168.203.115@tcp [ 7277.038506] Key type .llcrypt unregistered [ 7277.046859] Key type ._llcrypt unregistered [ 7321.025225] alg: No test for adler32 (adler32-zlib) [ 7321.788749] Key type ._llcrypt registered [ 7321.796442] Key type .llcrypt registered [ 7321.965245] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7340.239812] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7342.460367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7343.001319] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 7343.226738] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7343.233850] LNet: Accept secure, port 988 [ 7344.927101] Key type lgssc registered [ 7346.757440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7358.004507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7366.551492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7374.724044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7382.431225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7391.631259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7391.695304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.837683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7392.870290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7392.913106] Lustre: lustre-MDT0000: new disk, initializing [ 7392.961912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7392.981375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7398.271618] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7411.138778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7411.167839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.211130] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7411.262883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7411.275616] Lustre: Skipped 1 previous similar message [ 7411.391682] Lustre: lustre-MDT0001: new disk, initializing [ 7411.517928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7411.572901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7411.594818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7417.301771] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7428.624472] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7435.626921] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7444.251253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7444.352628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7444.367164] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7444.712237] Lustre: lustre-OST0000: new disk, initializing [ 7444.715604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7444.780748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.786336] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7454.324315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7454.339484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7461.154222] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7466.364282] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7466.681657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7472.506095] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7472.858822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7475.171937] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7475.176179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.177883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.193134] Lustre: Skipped 1 previous similar message [ 7480.291305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7480.312190] Lustre: Skipped 2 previous similar messages [ 7481.223086] Lustre: server umount lustre-OST0000 complete [ 7488.485624] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7488.495430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7488.531323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7490.530289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.534531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7490.583996] Lustre: Skipped 1 previous similar message [ 7495.649077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7495.661151] Lustre: Skipped 1 previous similar message [ 7501.279382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7501.463826] Lustre: server umount lustre-MDT0000 complete [ 7501.878751] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.911220] LustreError: Skipped 1 previous similar message [ 7506.438950] LustreError: 372713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774525149 with bad export cookie 15679756865094203296 [ 7506.455185] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.458194] LustreError: 372713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7507.157475] Lustre: server umount lustre-MDT0001 complete [ 7518.445905] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 07:39:19 (1774525159) [ 7526.028786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7533.011734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7559.831250] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7569.203856] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7580.303315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7588.641899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7595.917138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7603.005560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7610.323720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7610.471937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.826964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7610.894113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7610.987917] Lustre: lustre-MDT0000: new disk, initializing [ 7611.065979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.083964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7616.479700] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7627.967414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7628.047807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.120955] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7628.164663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7628.170248] Lustre: Skipped 1 previous similar message [ 7628.249870] Lustre: lustre-MDT0001: new disk, initializing [ 7628.334370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7628.391857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7628.404677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7632.847405] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7644.733564] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7650.546721] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7658.166173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7658.253156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7658.263866] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7658.542489] Lustre: lustre-OST0000: new disk, initializing [ 7658.550238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7658.665857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.769919] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7665.232670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7665.247498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7677.060966] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7683.397076] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7683.729812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7689.511451] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7689.684702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7696.352064] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7696.359198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.364713] LustreError: Skipped 1 previous similar message [ 7696.365575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.365579] Lustre: Skipped 2 previous similar messages [ 7696.420893] Lustre: Skipped 1 previous similar message [ 7697.729634] Lustre: server umount lustre-OST0000 complete [ 7705.056223] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7705.060658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7708.623191] Lustre: server umount lustre-MDT0000 complete [ 7711.721671] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.742047] LustreError: Skipped 1 previous similar message [ 7712.354584] LustreError: 380489:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774525355 with bad export cookie 15679756865094204150 [ 7712.355913] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7712.370756] LustreError: 380489:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7721.534867] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 07:42:43 (1774525363) [ 7731.341526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.686367] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7731.748604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.522763] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7736.810247] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.926965] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.362225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.780141] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7761.464918] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7767.975223] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7775.708118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7776.174193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.179680] Lustre: Skipped 1 previous similar message [ 7781.491333] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7790.361762] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7798.360522] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 7800.339925] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 7802.370746] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 7804.471651] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 7806.547511] Lustre: Failing over lustre-MDT0000 [ 7806.803071] Lustre: server umount lustre-MDT0000 complete [ 7806.805290] Lustre: Skipped 1 previous similar message [ 7806.952618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7806.968049] Lustre: Skipped 3 previous similar messages [ 7806.972573] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.995502] LustreError: Skipped 2 previous similar messages [ 7819.232275] Lustre: 370940:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774525455/real 1774525455] req@000000005a83db47 x1860724311777728/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774525462 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7819.261726] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7824.363465] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.390866] LustreError: Skipped 16 previous similar messages [ 7826.529967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.657882] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0xd999afcb3c6ebe4c to 0xd999afcb3c6ec23c [ 7836.679919] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [ 7837.001717] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7839.143125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7841.414396] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7842.331245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7842.363025] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7842.433261] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 7851.443206] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7853.884444] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7858.057844] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 7860.335177] Lustre: Failing over lustre-MDT0000 [ 7862.624463] Lustre: server umount lustre-MDT0000 complete [ 7862.770211] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.794403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7862.795731] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7862.812320] LustreError: Skipped 15 previous similar messages [ 7868.895290] Lustre: 370940:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774525505/real 1774525505] req@000000002c5e491b x1860724311788352/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774525512 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7868.914848] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.211454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.407741] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0xd999afcb3c6ec23c to 0xd999afcb3c6ec53e [ 7886.461898] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [ 7886.475079] Lustre: Skipped 2 previous similar messages [ 7886.883463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.892395] Lustre: Skipped 1 previous similar message [ 7886.966562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7888.190987] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7891.454612] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7891.956364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7891.980358] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7892.034377] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 7902.073545] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7903.653783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7912.417139] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.422963] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7912.427834] Lustre: Skipped 2 previous similar messages [ 7912.448573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7912.451921] Lustre: Skipped 6 previous similar messages [ 7915.957987] Lustre: server umount lustre-MDT0000 complete [ 7920.061843] LustreError: 389290:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774525563 with bad export cookie 15679756865094206782 [ 7920.072212] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7920.072855] LustreError: 389290:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7930.335139] Lustre: 390942:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774525567/real 1774525567] req@00000000cb7abfe2 x1860724311800960/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774525573 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7947.716392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.359253] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.371659] LustreError: Skipped 32 previous similar messages [ 7953.855752] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7964.031437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7969.651184] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7978.476232] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7984.739555] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7991.835770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7996.608272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7999.371506] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 8004.681380] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8009.696975] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8009.699238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.708109] LustreError: Skipped 1 previous similar message [ 8009.712954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.744795] Lustre: Skipped 2 previous similar messages [ 8011.729203] Lustre: server umount lustre-OST0000 complete [ 8011.731234] Lustre: Skipped 2 previous similar messages [ 8025.797619] LustreError: 391750:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774525668 with bad export cookie 15679756865094207517 [ 8025.801543] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8025.806972] LustreError: 391750:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8038.601035] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8041.672807] Key type lgssc unregistered [ 8041.976643] LNet: 395425:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8041.985467] LNet: Removed LNI 192.168.203.115@tcp [ 8042.890290] Key type .llcrypt unregistered [ 8042.894967] Key type ._llcrypt unregistered [ 8068.234057] alg: No test for adler32 (adler32-zlib) [ 8068.992501] Key type ._llcrypt registered [ 8068.995333] Key type .llcrypt registered [ 8069.125519] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8070.068965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8071.135583] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 8071.487938] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8071.499975] LNet: Accept secure, port 988 [ 8073.255778] Key type lgssc registered [ 8074.679474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8082.798707] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.808068] LustreError: 396910:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8082.820380] LustreError: 396910:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8082.826489] LustreError: 396910:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8082.833584] LustreError: 396910:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8082.847431] LustreError: 396910:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8090.780185] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8090.787912] LustreError: 397307:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8090.795728] LustreError: 397307:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8090.802522] LustreError: 397307:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8090.806972] LustreError: 397307:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8090.813593] LustreError: 397307:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8111.813697] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8114.792633] Key type lgssc unregistered [ 8115.111689] LNet: 398266:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8115.121258] LNet: Removed LNI 192.168.203.115@tcp [ 8115.909416] Key type .llcrypt unregistered [ 8115.912239] Key type ._llcrypt unregistered [ 8134.410170] alg: No test for adler32 (adler32-zlib) [ 8135.172675] Key type ._llcrypt registered [ 8135.177477] Key type .llcrypt registered [ 8135.290964] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8150.088426] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8151.278781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.695654] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 8151.845722] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8151.855312] LNet: Accept secure, port 988 [ 8153.504520] Key type lgssc registered [ 8154.485865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.138211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8170.148205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8176.545654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8183.616550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8191.543129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8191.595821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.894179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8192.946968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8193.064957] Lustre: lustre-MDT0000: new disk, initializing [ 8193.212079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8193.249753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8198.025225] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8209.012745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8209.127332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.278667] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8209.345911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8209.356332] Lustre: Skipped 1 previous similar message [ 8209.386269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8209.478578] Lustre: lustre-MDT0001: new disk, initializing [ 8209.634254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8209.715975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8209.729163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8215.918667] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8227.161104] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8234.425244] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8241.824024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8241.906813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.918957] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8242.241981] Lustre: lustre-OST0000: new disk, initializing [ 8242.249823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8242.358571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.511634] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8250.435289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8250.441172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8258.720409] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8264.257369] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8264.526878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8270.442352] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8270.624729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8276.452772] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8276.453874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.468907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8276.476093] Lustre: Skipped 1 previous similar message [ 8276.493583] Lustre: Skipped 1 previous similar message [ 8281.585190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8281.597051] Lustre: Skipped 1 previous similar message [ 8286.701590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.711403] Lustre: Skipped 1 previous similar message [ 8287.199284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8287.489814] Lustre: server umount lustre-OST0000 complete [ 8291.816402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.817347] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8291.818379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.834382] Lustre: Skipped 1 previous similar message [ 8297.765712] Lustre: server umount lustre-MDT0000 complete [ 8301.860766] LustreError: 402425:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774525944 with bad export cookie 9189831739071991165 [ 8301.871657] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8301.882861] LustreError: 402425:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8302.178539] Lustre: server umount lustre-MDT0001 complete [ 8312.268670] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8314.102810] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:52:35 (1774525955) [ 8321.716150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.127021] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8322.187310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8327.360588] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8327.651937] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.768702] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8335.705968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.061456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8341.115509] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8349.378941] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8355.878230] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.782690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.183533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.739547] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8375.809841] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8382.681088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8382.738275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8382.747495] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8382.854200] Lustre: lustre-OST0001: new disk, initializing [ 8382.859292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8382.955105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8387.473723] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8392.315289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8392.324133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8397.684467] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8402.912339] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8402.926446] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.937621] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8402.937630] Lustre: Skipped 3 previous similar messages [ 8402.973800] Lustre: Skipped 1 previous similar message [ 8407.828457] Lustre: server umount lustre-OST0001 complete [ 8408.043167] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8408.069114] LustreError: Skipped 1 previous similar message [ 8413.154940] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8413.181975] LustreError: Skipped 2 previous similar messages [ 8423.394537] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8423.424306] LustreError: Skipped 3 previous similar messages [ 8428.001205] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8428.012194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.033035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.036306] Lustre: Skipped 2 previous similar messages [ 8439.263201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8439.524412] Lustre: server umount lustre-OST0000 complete [ 8443.878412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.896037] Lustre: Skipped 1 previous similar message [ 8448.481599] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8449.844318] Lustre: server umount lustre-MDT0000 complete [ 8453.470935] LustreError: 409091:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774526096 with bad export cookie 9189831739071992026 [ 8453.475934] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8453.479197] LustreError: 409091:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8473.319819] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8476.142558] Key type lgssc unregistered [ 8476.371112] LNet: 411083:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8476.377333] LNet: Removed LNI 192.168.203.115@tcp [ 8477.179187] Key type .llcrypt unregistered [ 8477.181058] Key type ._llcrypt unregistered [ 8515.174291] alg: No test for adler32 (adler32-zlib) [ 8515.934195] Key type ._llcrypt registered [ 8515.935777] Key type .llcrypt registered [ 8516.058992] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8530.884600] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8532.139858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8533.046136] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 8533.269182] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8533.276383] LNet: Accept secure, port 988 [ 8534.919119] Key type lgssc registered [ 8535.941404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8544.607860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8551.314183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8557.289565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8564.574611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8572.159723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8572.214700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.443133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8573.468755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8573.545770] Lustre: lustre-MDT0000: new disk, initializing [ 8573.607239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.622849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8578.859625] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8590.284164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8590.343495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.395856] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8590.422654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8590.426809] Lustre: Skipped 1 previous similar message [ 8590.508645] Lustre: lustre-MDT0001: new disk, initializing [ 8590.597624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8590.642321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8590.651815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8595.221682] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8606.957571] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8613.626485] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8620.991184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8621.062587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8621.068750] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8621.308441] Lustre: lustre-OST0000: new disk, initializing [ 8621.312384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8621.420903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.280678] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8636.818323] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8641.789732] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8641.981967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8646.718393] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8646.899969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8650.209306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8650.226544] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8650.226571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8650.226879] Lustre: Skipped 1 previous similar message [ 8655.239660] Lustre: server umount lustre-OST0000 complete [ 8660.448405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.451766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.468833] Lustre: Skipped 1 previous similar message [ 8660.487961] Lustre: Skipped 2 previous similar messages [ 8665.568864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.574413] Lustre: Skipped 1 previous similar message [ 8665.632942] Lustre: server umount lustre-MDT0000 complete [ 8666.593132] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8666.610410] LustreError: Skipped 1 previous similar message [ 8669.598345] LustreError: 415198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774526312 with bad export cookie 9344118980810490897 [ 8669.615821] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.618143] LustreError: 415198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8669.975914] Lustre: server umount lustre-MDT0001 complete [ 8680.220987] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 07:58:41 (1774526321) [ 8690.345902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8699.795786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.374790] Lustre: lustre-MDT0000: new disk, initializing [ 8700.424190] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.481922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.405235] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8705.506929] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.635576] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.963304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.372318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.051489] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8730.071867] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8736.148265] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8740.833966] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8740.847493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.869569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8744.579662] Lustre: server umount lustre-MDT0000 complete [ 8744.930525] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.944036] LustreError: Skipped 1 previous similar message [ 8748.864927] LustreError: 420330:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774526391 with bad export cookie 9344118980810491751 [ 8748.875318] LustreError: 420330:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8748.892558] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.165305] Lustre: server umount lustre-MDT0001 complete [ 8758.908227] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:00:00 (1774526400) [ 8767.113057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.652024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8767.745903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8774.066811] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8784.493729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.797132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8789.559865] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8797.793787] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8804.203608] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8811.259368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8814.666618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8814.685227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8817.049841] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8826.499108] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8833.967708] Lustre: Permanently deactivating lustre-OST0000 [ 8833.981972] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8844.333304] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8845.759838] LustreError: 426058:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 8845.771210] LustreError: 426058:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8845.776411] LustreError: 426058:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8849.357686] LustreError: 426106:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8849.362245] LustreError: 426106:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8857.644204] Lustre: server umount lustre-OST0000 complete [ 8860.317635] LustreError: 426356:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8860.327048] LustreError: 426356:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8866.272338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8866.285262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8866.306924] Lustre: Skipped 2 previous similar messages [ 8866.319659] Lustre: Skipped 3 previous similar messages [ 8868.831174] Lustre: 426453:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774526505/real 1774526505] req@0000000086764570 x1860725559575040/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774526511 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8871.405160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.420108] LustreError: Skipped 4 previous similar messages [ 8873.716846] LustreError: 422933:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774526516 with bad export cookie 9344118980810492304 [ 8873.724983] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8873.735337] LustreError: 422933:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8879.071168] Lustre: 426655:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774526516/real 1774526516] req@000000009d801c5b x1860725559576064/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774526522 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8879.557422] Lustre: server umount lustre-MDT0001 complete [ 8879.563540] Lustre: Skipped 1 previous similar message [ 8887.961318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.489280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8895.265991] LustreError: 427566:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 8901.450406] Lustre: 427912:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.115@tcp [ 8909.791258] Lustre: 428011:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774526546/real 1774526546] req@00000000d6d0d9cc x1860725559576576/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774526552 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8921.951361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.371952] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.396456] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8922.404153] Lustre: Skipped 1 previous similar message [ 8922.466362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.469226] Lustre: Skipped 1 previous similar message [ 8927.106365] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8935.868172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.216994] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8941.199300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8948.846951] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8960.632238] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8963.871493] Lustre: Permanently reactivating lustre-OST0000 [ 8963.884787] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8963.890278] Lustre: Skipped 2 previous similar messages [ 8992.140509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8992.551811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.554639] Lustre: Skipped 1 previous similar message [ 8998.853900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9008.538114] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9018.338987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.363495] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9018.369803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9021.973581] Lustre: server umount lustre-OST0000 complete [ 9021.976055] Lustre: Skipped 1 previous similar message [ 9028.576583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.586671] Lustre: Skipped 2 previous similar messages [ 9033.700941] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.715427] LustreError: Skipped 3 previous similar messages [ 9038.284242] LustreError: 428620:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774526681 with bad export cookie 9344118980810493256 [ 9038.290086] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9038.309991] LustreError: 428620:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9051.992685] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9055.140316] Key type lgssc unregistered [ 9055.619139] LNet: 432742:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9055.624710] LNet: Removed LNI 192.168.203.115@tcp [ 9056.679369] Key type .llcrypt unregistered [ 9056.681938] Key type ._llcrypt unregistered [ 9079.864541] alg: No test for adler32 (adler32-zlib) [ 9080.620588] Key type ._llcrypt registered [ 9080.623644] Key type .llcrypt registered [ 9080.793630] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 9097.861683] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9100.013214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9100.608352] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 9100.935045] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9100.939791] LNet: Accept secure, port 988 [ 9102.751283] Key type lgssc registered [ 9104.150688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9115.070313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9123.398725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9131.276361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9138.727310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9145.878541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9145.964559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.179920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9147.216858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9147.343330] Lustre: lustre-MDT0000: new disk, initializing [ 9147.470227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.482448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9152.177796] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9163.261082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9163.359600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.488266] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9163.537402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9163.542408] Lustre: Skipped 1 previous similar message [ 9163.652832] Lustre: lustre-MDT0001: new disk, initializing [ 9163.786797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9163.841064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9163.852771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9168.684760] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9179.260882] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9185.009200] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9191.734047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9191.875848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.886195] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9192.331508] Lustre: lustre-OST0000: new disk, initializing [ 9192.336336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9192.446676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.995849] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9201.186913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9201.203677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9207.352443] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9212.395707] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9212.603835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9218.346115] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9218.562143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9221.600655] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9221.610112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.637631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.113402] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.116634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.139388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.144097] Lustre: Skipped 1 previous similar message [ 9226.576974] Lustre: server umount lustre-OST0000 complete [ 9232.357173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.367103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9237.472267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9237.479268] Lustre: Skipped 2 previous similar messages [ 9242.600610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.611581] Lustre: Skipped 1 previous similar message [ 9245.151149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9245.270584] Lustre: server umount lustre-MDT0000 complete [ 9248.742750] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.760897] LustreError: Skipped 1 previous similar message [ 9249.480825] LustreError: 436034:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774526892 with bad export cookie 10040994665687216497 [ 9249.485472] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9249.487284] LustreError: 436034:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9249.768974] Lustre: server umount lustre-MDT0001 complete [ 9258.965498] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9260.833312] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:08:22 (1774526902) [ 9267.857317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9268.180937] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9268.247604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9273.033430] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9273.324376] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.433423] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9282.050995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.550287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9287.586149] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9296.834750] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9303.705927] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9310.358789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9310.775507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9315.442660] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9323.872954] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9326.018630] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9329.476316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9338.349221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.371358] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9338.373046] Lustre: Skipped 2 previous similar messages [ 9338.375286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9338.375293] Lustre: Skipped 2 previous similar messages [ 9344.542928] Lustre: server umount lustre-OST0000 complete [ 9353.705898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9355.580370] Lustre: server umount lustre-MDT0000 complete [ 9358.819524] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.836026] LustreError: Skipped 1 previous similar message [ 9359.441794] LustreError: 440630:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774527002 with bad export cookie 10040994665687217344 [ 9359.446632] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.448548] LustreError: 440630:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9373.174262] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9377.190288] Key type lgssc unregistered [ 9377.647664] LNet: 444497:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9377.654420] LNet: Removed LNI 192.168.203.115@tcp [ 9378.696249] Key type .llcrypt unregistered [ 9378.697903] Key type ._llcrypt unregistered [ 9406.129267] alg: No test for adler32 (adler32-zlib) [ 9406.885041] Key type ._llcrypt registered [ 9406.887956] Key type .llcrypt registered [ 9407.005986] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9407.268625] Key type .llcrypt unregistered [ 9407.270815] Key type ._llcrypt unregistered [ 9425.556299] alg: No test for adler32 (adler32-zlib) [ 9426.312205] Key type ._llcrypt registered [ 9426.316089] Key type .llcrypt registered [ 9426.424048] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9428.010480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9428.598411] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 9428.931041] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9428.937354] LNet: Accept secure, port 988 [ 9430.655109] Key type lgssc registered [ 9432.084246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9440.407092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.917365] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.979764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9446.255456] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9447.407160] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9452.518486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.105479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.389898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.418795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9457.466385] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 9460.364942] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9462.757579] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.772676] LustreError: Skipped 4 previous similar messages [ 9468.356949] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9472.997396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9473.016962] LustreError: Skipped 3 previous similar messages [ 9475.964874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.390153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9481.495651] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9491.898139] Lustre: server umount lustre-OST0000 complete [ 9497.057208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.062046] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9497.075294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9499.103147] Lustre: 446163:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774527135/real 1774527135] req@000000009a4e49ef x1860726499062400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774527142 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9499.107677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9499.137952] Lustre: 446163:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9499.161833] Lustre: Skipped 1 previous similar message [ 9499.170330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9499.181133] Lustre: Skipped 2 previous similar messages [ 9501.151153] Lustre: 449310:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774527138/real 1774527138] req@00000000c4c257f3 x1860726499063040/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774527144 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9501.567758] Lustre: server umount lustre-MDT0000 complete [ 9504.223196] Lustre: 446163:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774527140/real 1774527140] req@0000000097091eb9 x1860726499063296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774527147 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9504.232892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.275346] LustreError: Skipped 1 previous similar message [ 9505.695787] LustreError: 446829:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774527148 with bad export cookie 3480023768212720337 [ 9505.707709] LustreError: 446829:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9505.710193] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9506.058904] Lustre: server umount lustre-MDT0001 complete [ 9519.436587] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9521.886953] Key type lgssc unregistered [ 9522.145907] LNet: 450116:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9522.158877] LNet: Removed LNI 192.168.203.115@tcp [ 9522.886752] Key type .llcrypt unregistered [ 9522.889574] Key type ._llcrypt unregistered [ 9546.259645] alg: No test for adler32 (adler32-zlib) [ 9547.014332] Key type ._llcrypt registered [ 9547.018133] Key type .llcrypt registered [ 9547.144296] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9548.148848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9548.831810] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 9549.053072] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9549.064496] LNet: Accept secure, port 988 [ 9550.751414] Key type lgssc registered [ 9551.905850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9560.435601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.119373] LustreError: 451552:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 1: rc = -5 [ 9608.863531] LustreError: 451552:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 9639.647394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.344777] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9652.772141] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9660.481374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9685.836473] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.904458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.963709] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 9691.108975] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9691.519852] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9696.234640] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9701.352791] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.028359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9702.591653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.042464] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9716.466389] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9723.149480] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9732.064479] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9732.075096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.084839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9733.090509] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9733.108469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9733.128169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9734.630297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9734.637148] Lustre: Skipped 1 previous similar message [ 9736.415946] Lustre: server umount lustre-OST0000 complete [ 9743.329962] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9743.346334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.358855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9746.742119] Lustre: server umount lustre-MDT0000 complete [ 9749.991861] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.015366] LustreError: Skipped 1 previous similar message [ 9751.156984] LustreError: 451561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774527394 with bad export cookie 8218367461805627060 [ 9751.157405] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9751.166803] LustreError: 451561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9751.434027] Lustre: server umount lustre-MDT0001 complete [ 9765.230798] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9768.146878] Key type lgssc unregistered [ 9768.437372] LNet: 455251:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9768.449746] LNet: Removed LNI 192.168.203.115@tcp [ 9769.341252] Key type .llcrypt unregistered [ 9769.343887] Key type ._llcrypt unregistered [ 9796.161927] alg: No test for adler32 (adler32-zlib) [ 9796.919533] Key type ._llcrypt registered [ 9796.922347] Key type .llcrypt registered [ 9797.048508] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9798.751835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9799.337709] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 9799.643217] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9799.650438] LNet: Accept secure, port 988 [ 9801.360851] Key type lgssc registered [ 9802.595494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9811.550905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9812.932538] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.000269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9817.078880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9818.095507] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9823.202900] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.149872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.473972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9830.091435] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9837.566414] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9843.798551] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9850.481808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9850.842103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9856.150304] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9864.232768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 9864.443567] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9869.281726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9869.285192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9869.288772] Lustre: Skipped 1 previous similar message [ 9871.181074] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 9871.193482] Lustre: Skipped 2 previous similar messages [ 9874.404828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9874.413278] Lustre: Skipped 2 previous similar messages [ 9874.885883] Lustre: server umount lustre-MDT0000 complete [ 9876.294710] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9878.089600] LustreError: 456717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.15@tcp arrived at 1774527521 with bad export cookie 17765414427569657286 [ 9880.334449] LustreError: 456717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774527523 with bad export cookie 17765414427569656390 [ 9880.351032] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9880.357100] LustreError: 456717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9880.545141] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9880.556635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.570996] Lustre: Skipped 1 previous similar message [ 9880.582028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.641030] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.643577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.653065] LustreError: Skipped 4 previous similar messages [ 9886.679884] Lustre: server umount lustre-OST0000 complete [ 9893.434832] Lustre: server umount lustre-MDT0001 complete [ 9905.660290] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9908.209761] Key type lgssc unregistered [ 9908.474895] LNet: 460545:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9908.484445] LNet: Removed LNI 192.168.203.115@tcp [ 9909.173740] Key type .llcrypt unregistered [ 9909.179248] Key type ._llcrypt unregistered [ 9932.015958] alg: No test for adler32 (adler32-zlib) [ 9932.779428] Key type ._llcrypt registered [ 9932.781616] Key type .llcrypt registered [ 9932.928138] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9934.499914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9935.037743] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [ 9935.337098] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9935.348453] LNet: Accept secure, port 988 [ 9937.039120] Key type lgssc registered [ 9938.194360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9945.879562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9947.195413] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.233375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.938574] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9952.738262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.858425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9961.554415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.034350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9966.427736] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9974.268208] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9980.490178] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9988.179266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.649379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9993.024691] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9997.935757] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [10002.121944] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10008.563677] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.589903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10008.599301] Lustre: Skipped 1 previous similar message [10009.437597] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [10012.582707] Lustre: server umount lustre-MDT0001 complete [10013.686303] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.705186] LustreError: Skipped 1 previous similar message [10018.793424] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10018.816442] LustreError: Skipped 2 previous similar messages [10023.904908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.919855] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10023.933763] Lustre: Skipped 1 previous similar message [10023.940972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10025.148833] Lustre: server umount lustre-OST0000 complete [10035.167203] Lustre: 465061:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774527672/real 1774527672] req@000000006f512a0b x1860727029644800/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774527678 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10035.440932] Lustre: server umount lustre-MDT0000 complete [10051.480684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10053.924369] Key type lgssc unregistered [10054.216192] LNet: 465817:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10054.233441] LNet: Removed LNI 192.168.203.115@tcp [10054.962654] Key type .llcrypt unregistered [10054.965637] Key type ._llcrypt unregistered [10080.620325] alg: No test for adler32 (adler32-zlib) [10081.382202] Key type ._llcrypt registered [10081.384527] Key type .llcrypt registered [10081.519757] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10094.154442] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10094.969377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10095.631463] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [10095.841923] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10095.848720] LNet: Accept secure, port 988 [10097.578593] Key type lgssc registered [10098.533763] Lustre: Echo OBD driver; http://www.lustre.org/ [10107.536325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10114.792700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10120.314146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10126.436355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10133.999901] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10140.708510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10146.289668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10146.343116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10147.537522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10147.574484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10147.681329] Lustre: lustre-MDT0000: new disk, initializing [10147.771601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10147.781852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10151.640306] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10158.558444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10163.648109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10163.708904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10163.714337] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10163.883696] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [10163.891135] Lustre: lustre-OST0000: new disk, initializing [10163.900240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10163.902763] Lustre: Skipped 1 previous similar message [10163.954393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10167.615500] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10176.195573] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10183.621629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10190.213836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10190.313468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.452551] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10190.523987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10190.647813] Lustre: lustre-MDT0001: new disk, initializing [10190.727436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10190.737750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10190.791791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10195.927967] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10204.838684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [10205.039238] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10209.289377] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [10209.443955] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10217.973513] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.979217] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10217.997494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10218.009235] Lustre: Skipped 1 previous similar message [10220.338644] Lustre: server umount lustre-MDT0001 complete [10223.079898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.087300] LustreError: Skipped 1 previous similar message [10228.193840] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.223181] LustreError: Skipped 2 previous similar messages [10233.316197] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.334957] LustreError: Skipped 1 previous similar message [10238.453099] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10238.455182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10238.484902] Lustre: Skipped 1 previous similar message [10238.487458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10238.492149] Lustre: Skipped 1 previous similar message [10243.552147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10243.555850] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.673920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10251.231121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10251.368650] Lustre: server umount lustre-OST0000 complete [10253.793340] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.803919] LustreError: Skipped 1 previous similar message [10267.619698] Lustre: 474232:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774527904/real 1774527904] req@00000000fd1193f9 x1860727197421376/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774527910 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10267.789660] Lustre: server umount lustre-MDT0000 complete [10280.926397] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10283.352514] Key type lgssc unregistered [10283.593468] LNet: 474988:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10283.606278] LNet: Removed LNI 192.168.203.115@tcp [10284.220835] Key type .llcrypt unregistered [10284.223254] Key type ._llcrypt unregistered [10314.701737] alg: No test for adler32 (adler32-zlib) [10315.459663] Key type ._llcrypt registered [10315.462063] Key type .llcrypt registered [10315.607274] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10315.917617] Key type .llcrypt unregistered [10315.921700] Key type ._llcrypt unregistered [10329.002679] alg: No test for adler32 (adler32-zlib) [10329.755834] Key type ._llcrypt registered [10329.761333] Key type .llcrypt registered [10329.906195] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10330.824069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.105448] Lustre: Lustre: Build Version: 2.15.8_1_g13eca2c [10331.270249] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10331.277042] LNet: Accept secure, port 988 [10332.919316] Key type lgssc registered [10333.806647] Lustre: Echo OBD driver; http://www.lustre.org/ [10340.397452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10345.525474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10350.051656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10355.887436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10355.937812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.159600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10357.178819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10357.237446] Lustre: lustre-MDT0000: new disk, initializing [10357.287417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10357.301903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10360.772576] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10369.470479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10369.512976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.561193] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10369.589140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10369.592470] Lustre: Skipped 1 previous similar message [10369.658369] Lustre: lustre-MDT0001: new disk, initializing [10369.733735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10369.762483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10369.774291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10373.695776] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10381.673779] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10386.391390] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10391.835485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10391.901772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10391.908356] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10392.088209] Lustre: lustre-OST0000: new disk, initializing [10392.091269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10392.160133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10395.949856] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10400.313344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10400.325643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10404.214468] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10410.985194] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10410.986653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10410.993790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10411.002935] Lustre: Skipped 2 previous similar messages [10411.013809] Lustre: Skipped 2 previous similar messages [10411.912403] Lustre: server umount lustre-MDT0000 complete [10415.406376] LustreError: 480565:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774528058 with bad export cookie 11708715062487761333 [10415.418103] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10415.421890] LustreError: 480565:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10415.750778] Lustre: server umount lustre-MDT0001 complete [10425.311423] Lustre: 481731:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774528062/real 1774528062] req@00000000cd7a6a16 x1860727444881280/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774528068 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10425.620069] Lustre: server umount lustre-OST0000 complete [10433.617525] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 08:27:55 (1774528075) [10434.359328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10439.623641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.952885] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.007957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10443.910488] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10445.282723] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.415036] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.255090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.556450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10455.598463] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10463.036818] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10467.633686] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10473.531046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10473.617464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.736040] Lustre: Found index 0 for lustre-OST0000, updating log [10473.859882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10478.001598] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10484.634346] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10490.290474] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [10500.580337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.582716] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10500.597530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10500.605986] Lustre: Skipped 1 previous similar message [10505.611301] Lustre: server umount lustre-OST0000 complete [10510.816640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.818550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10510.834672] Lustre: Skipped 2 previous similar messages [10515.269355] Lustre: server umount lustre-MDT0000 complete [10515.939080] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10515.944361] LustreError: Skipped 1 previous similar message [10518.648808] LustreError: 485143:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774528161 with bad export cookie 11708715062487762677 [10518.658303] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.667434] LustreError: 485143:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [10531.807309] LustreError: 485646:0:(import.c:355:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) [10531.811106] LustreError: 485646:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000f8ee1b6d x1860727444899008/t0(0) o8->lustre-OST0000-osc-MDT0001@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1774528171 ref 2 fl UnregRPC:ENU/0/ffffffff rc -5/-1 job:'' [10531.845369] LustreError: 485646:0:(import.c:389:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. [10533.293465] Lustre: server umount lustre-MDT0001 complete [10540.125386] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 08:29:42 (1774528182) [10544.587037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10547.961681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10552.029598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10556.023469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10568.501362] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10575.448632] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10583.174637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10588.846878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10593.460526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10599.143523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10604.472902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10604.545179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10604.707156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10604.728119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10604.796836] Lustre: lustre-MDT0000: new disk, initializing [10604.853644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10604.865512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10608.730782] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10617.686624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10617.753033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.805228] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10617.828079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10617.831479] Lustre: Skipped 1 previous similar message [10617.897528] Lustre: lustre-MDT0001: new disk, initializing [10617.989837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10618.002611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10622.367193] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10631.418463] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10636.679778] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10642.280799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10642.358969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10642.370790] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10642.675858] Lustre: lustre-OST0000: new disk, initializing [10642.680038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10642.735869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10642.739016] Lustre: Skipped 1 previous similar message [10646.200243] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10652.734171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10652.741194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10654.568597] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10658.211813] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10658.381979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10662.242774] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10662.390857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10668.514943] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10668.517468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.521939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10668.534302] Lustre: Skipped 1 previous similar message [10668.544287] Lustre: Skipped 2 previous similar messages [10673.635147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10673.643018] Lustre: Skipped 1 previous similar message [10678.239351] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10678.382649] Lustre: server umount lustre-OST0000 complete [10683.877236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.879814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.883911] Lustre: Skipped 1 previous similar message [10683.890424] Lustre: Skipped 1 previous similar message [10688.992201] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.014059] LustreError: Skipped 3 previous similar messages [10690.645688] LustreError: 489601:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774528333 with bad export cookie 11708715062487763748 [10690.649290] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10690.653564] LustreError: 489601:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10710.560838] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10718.025627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10727.932935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10735.793118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10741.264865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10747.077355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10748.524409] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10505 sec ========= 08:33:10 (1774528390) [10778.462576] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10780.259557] Key type lgssc unregistered [10780.451553] LNet: 498339:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10780.458755] LNet: Removed LNI 192.168.203.115@tcp [10781.030603] Key type .llcrypt unregistered [10781.033207] Key type ._llcrypt unregistered